Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Fork Choice compliance test generator #3831

Draft
wants to merge 113 commits into
base: dev
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
113 commits
Select commit Hold shift + click to select a range
68deb2d
Implement basic version of SM link generator
mkalinin Dec 12, 2023
f989f37
Remove numpy as a dependency
mkalinin Dec 18, 2023
967445f
Clean up generator's code
mkalinin Dec 18, 2023
2101e9f
Add debug output
mkalinin Dec 18, 2023
507bac8
Add anchor_epoch as a parameter and debug checks
mkalinin Dec 18, 2023
5885bf4
Switch sm_links generator to minizinc model
mkalinin Dec 19, 2023
0c7ff17
Rename sm_links test and its deps
mkalinin Dec 21, 2023
a6e9dd5
Disable debug mode
mkalinin Dec 21, 2023
e00346a
Fixes and polishing
mkalinin Dec 21, 2023
cf38551
Update sm_links test and generator
mkalinin Dec 28, 2023
ae322c7
Fix contraint reachability problem
mkalinin Dec 28, 2023
3a2d5fe
Update debug output
mkalinin Dec 28, 2023
995cab2
Add CLI args to fork choice generator
mkalinin Dec 29, 2023
beac058
Fix bugs, inrease verbosity
mkalinin Feb 13, 2024
e07f580
Add basic block tree model instantiator
mkalinin Feb 15, 2024
aab9f4b
Minor fixes
mkalinin Feb 27, 2024
c46b9a3
Add block cover instantiator
mkalinin Mar 14, 2024
84c658b
Update block_cover3 model and fix instantiator
mkalinin Mar 29, 2024
e4b4556
Sort tips before sampling
ericsson49 Apr 1, 2024
71e70ad
Added an option to pre-generated model instances: `--fc-gen-instances…
ericsson49 Apr 1, 2024
0f358da
Sample yaml file with pre-generated instances
ericsson49 Apr 1, 2024
c819ef4
minor fixes
ericsson49 Apr 1, 2024
9b5f6ec
rename params
ericsson49 Apr 1, 2024
a0999cd
Modified `filter_block_tree_generator.py`: added an option to loa pre…
ericsson49 Apr 1, 2024
5f00017
Implement on_attestation
mkalinin Apr 6, 2024
310893c
Implement attester slashings
mkalinin Apr 7, 2024
cc852ca
Update block tree model
mkalinin Apr 8, 2024
d477b33
Added `test_gen.py` driver to generate all tests from a single confi.
ericsson49 Apr 2, 2024
b0be1ae
Implement invalid blocks
mkalinin Apr 8, 2024
b563381
Omit valid: true for on_attestation
mkalinin Apr 9, 2024
e59e612
Mutation operators/shuffling added.
ericsson49 Apr 9, 2024
cedc82f
Added customizeable test_name.
ericsson49 Apr 10, 2024
b3dae8f
Pass block parents as a parameter, update block tree model
mkalinin Apr 10, 2024
3eda0b7
Added command line args for `with_attester_slashings` and `with_inval…
ericsson49 Apr 10, 2024
67138ef
Refactor block parents parametrization
mkalinin Apr 10, 2024
54756e9
Modified solutions construction
ericsson49 Apr 10, 2024
f4886e0
Changed block_tree format
ericsson49 Apr 10, 2024
4058856
Added test instances generator (from MiniZinc models to yaml)
ericsson49 Apr 10, 2024
8cc662b
Add invalid attestations and slashings
mkalinin Apr 11, 2024
12c214e
Handle slashed proposer as it were a regular invalid block
mkalinin Apr 11, 2024
573b5ef
Fix valid block production
mkalinin Apr 11, 2024
1f80417
fix solutions construction
ericsson49 Apr 11, 2024
be1b1f8
fixed file format (`sm_links` instead of `sources/targets`)
ericsson49 Apr 11, 2024
db5b03f
renamed GENERATOR_NAME
ericsson49 Apr 11, 2024
d4b33cb
fixed solution constuction (`sm_links` instead of `sources/targets`)
ericsson49 Apr 11, 2024
84ca475
Added a command line option to pass the test_gen configuration path
ericsson49 Apr 16, 2024
991d026
Ensure voting source matches the attestation source
mkalinin Apr 22, 2024
40aadb3
Remove over constraining assert check
mkalinin Apr 22, 2024
c4ca1bc
Ensure that epoch 1 is not supposed to be justified
mkalinin Apr 23, 2024
4c9e402
Re-formatted test_gen config
ericsson49 Apr 23, 2024
c6a1496
Refactoring of test generator routines: streamline code organization
ericsson49 Apr 23, 2024
6bf265a
Improve block cover instantiator to accommodate for more blocks
mkalinin Apr 24, 2024
0ff437a
Fix unexpected justifed checkpoint in block cover
mkalinin Apr 25, 2024
00abe77
Make block cover instantiator more flexible
mkalinin Apr 25, 2024
3c5ca9d
Make justified epoch 1 check stricter
mkalinin Apr 26, 2024
8b34ac8
Fixed bug in `is_leaf``
ericsson49 Apr 25, 2024
3ad01e4
temporary small refactoring
ericsson49 May 1, 2024
d2e029b
refactoring
ericsson49 May 1, 2024
d1dd31d
Extend FC checks with filtered block weights
mkalinin May 10, 2024
371288c
Move instantiators into a separate dir
mkalinin May 10, 2024
faee7d5
Unify test case instantiator
mkalinin May 11, 2024
fe5b078
new instance configurations added: small, smoke, standard
ericsson49 May 15, 2024
909a9f5
new testgen configs added: small, smoke and standard
ericsson49 May 15, 2024
51e68f0
add block_cover instances
ericsson49 May 15, 2024
2a352b0
add block_tree_tree instances
ericsson49 May 15, 2024
255f01e
add block_tree_other instances
ericsson49 May 15, 2024
50df903
fix typos
ericsson49 May 15, 2024
6f08518
put test configs into separate directories
ericsson49 May 15, 2024
6af67eb
Fix test DNA file paths to run test_gen from root dir
mkalinin May 29, 2024
6af1e9e
Use pyspec_tests suite name for fc generated tests
mkalinin May 29, 2024
c428779
Add bls_setting to meta data for fc generated tests
mkalinin May 29, 2024
f3a7194
Add pseudo randao_reveal for fc generated tests
mkalinin May 29, 2024
b049f90
Replace filtered_block_weights with viable_for_head_roots_and_weights
mkalinin May 29, 2024
de42529
Fix pseudo randao_reveal computation
mkalinin May 29, 2024
a6af078
Make MutationGenerator serializeable
ericsson49 May 30, 2024
f99592b
Make TestProviders finer granular
ericsson49 May 30, 2024
789704f
Fixed typo in standard/block_cover config
ericsson49 May 30, 2024
22b2a90
Fixed a problem with running in multiprocessor node - incompatible re…
ericsson49 Jun 11, 2024
a2c0997
Added shuffling_test (by splitting block_weight_test)
ericsson49 Jun 11, 2024
bbdae54
renamed small -> tiny, smoke -> small
ericsson49 Jun 11, 2024
bd5e38b
renamed small -> tiny, smoke -> small
ericsson49 Jun 11, 2024
e6505df
renamed small -> tiny, smoke -> small
ericsson49 Jun 11, 2024
399fd16
Refactoring to improve code structure and ensure deterministic behavi…
ericsson49 Jun 12, 2024
bb56274
Code restructuring
ericsson49 Jun 12, 2024
c9c345b
refactored code, to use test data from instantiators, instead of full…
ericsson49 Jun 18, 2024
638642b
Code reorg: make helper methods
ericsson49 Jun 19, 2024
fd21a46
split `yield_fork_choice_test_case` into two steps:
ericsson49 Jun 19, 2024
aa91e8d
refactoring of mutated test generation
ericsson49 Jun 19, 2024
7fe26e6
re-worked test generation: skip duplicate tick events
ericsson49 Jun 20, 2024
8233c8c
Links added:
ericsson49 Jun 25, 2024
05bbbc7
Test runner added
ericsson49 Jun 27, 2024
3cec3f1
Typos fixed - increase amount of tests for shuffling test
ericsson49 Jun 27, 2024
c947d0c
Fixed some bugs with scheduling
ericsson49 Jun 27, 2024
7c9ddbb
Move mutation_operators to instantiators
ericsson49 Jun 27, 2024
f184254
Extend fork choice test format
mkalinin Jun 28, 2024
47199a3
remove with_viable_for_head_weights=True (kept by mistake)
ericsson49 Jun 28, 2024
02a0287
put back MutationOps (dropped by mistake)
ericsson49 Jun 28, 2024
423515e
switch to DENEB as the default fork
ericsson49 Jun 28, 2024
9930464
remove debug prints and recovery property from test steps
ericsson49 Jul 1, 2024
6a27e1e
drop unnecessary comments
ericsson49 Jul 1, 2024
5a14ef6
Drop `checks_with_viable_for_head_weights` which is not a part of `on…
ericsson49 Jul 1, 2024
8545060
Merge pull request #17 from ericsson49/fc-compliance2
mkalinin Jul 1, 2024
4ed16ab
added `viable_for_head_roots_and_weights` check description
ericsson49 Jul 1, 2024
fb3ff0f
fixed names (small -> tiny, smoke -> small)
ericsson49 Jul 1, 2024
c494a96
added instructions how to generate and run tests
ericsson49 Jul 1, 2024
fb231de
Update README.md
ericsson49 Jul 1, 2024
eeafb1c
Update README.md
ericsson49 Jul 1, 2024
959274f
Update tests/formats/fork_choice/README.md
ericsson49 Jul 2, 2024
ce897c8
Fix viable_for_head_roots_and_weights in example
mkalinin Jul 2, 2024
4a0745b
Merge pull request #19 from ericsson49/fc-compliance2
mkalinin Jul 2, 2024
543ec86
Added a line about EF grant
ericsson49 Jul 3, 2024
1854266
Changed viable_for_head_roots_and_weights format
ericsson49 Jul 3, 2024
2eb1575
Added CLI switch to turn on multiprocessing mode
ericsson49 Jan 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Unify test case instantiator
mkalinin committed Jun 28, 2024
commit faee7d5a8243a2603d48bf7bf34ad7b44be57f67
Original file line number Diff line number Diff line change
@@ -3,11 +3,6 @@
spec_state_test,
with_altair_and_later,
)
from eth2spec.test.helpers.fork_choice import (
on_tick_and_append_step,
tick_and_add_block,
output_store_checks,
)
from eth2spec.test.helpers.state import (
transition_to,
next_slot,
@@ -21,6 +16,7 @@
advance_state_to_anchor_epoch,
produce_block,
attest_to_slot,
yield_fork_choice_test_case,
)

def _should_justify_epoch(parents, current_justifications, previous_justifications, block) -> bool:
@@ -222,23 +218,20 @@ def gen_block_cover_test_data(spec, state, model_params, debug, seed) -> (FCTest

blocks = [ProtocolMessage(block) for block in signed_blocks]

test_data = FCTestData(meta, anchor_block, anchor_state, blocks, [], [])
current_epoch_slot = spec.compute_start_slot_at_epoch(model_params['current_epoch'])
current_epoch_time = state.genesis_time + current_epoch_slot * spec.config.SECONDS_PER_SLOT

test_data = FCTestData(meta, anchor_block, anchor_state, blocks, store_final_time=current_epoch_time)
target_block_root = spec.hash_tree_root(post_block_tips[target_block].beacon_state.latest_block_header)

return test_data, target_block_root


def apply_final_steps(spec, store, model_params, test_steps):
current_epoch_slot = spec.compute_start_slot_at_epoch(model_params['current_epoch'])
current_epoch_time = store.genesis_time + current_epoch_slot * spec.config.SECONDS_PER_SLOT
if store.time < current_epoch_time:
on_tick_and_append_step(spec, store, current_epoch_time, test_steps)


def run_sanity_checks(spec, store, model_params, target_block_root):
current_epoch = spec.get_current_store_epoch(store)
# Ensure the epoch is correct
assert current_epoch == model_params['current_epoch']
assert current_epoch == model_params['current_epoch'], str(current_epoch) + ' != ' + str(
model_params['current_epoch'])
# Ensure the store.justified_checkpoint.epoch is as expected
assert store.justified_checkpoint.epoch == model_params['store_justified_epoch']

@@ -277,33 +270,7 @@ def run_sanity_checks(spec, store, model_params, target_block_root):
@spec_state_test
def yield_block_cover_test_case(spec, state, model_params=None, debug=False, seed=1):
test_data, target_block_root = gen_block_cover_test_data(spec, state, model_params, debug, seed)

# Yield meta
for k, v in test_data.meta.items():
yield k, 'meta', v

# Yield anchor state and block initialization
yield 'anchor_state', test_data.anchor_state
yield 'anchor_block', test_data.anchor_block

test_steps = []
store = spec.get_forkchoice_store(test_data.anchor_state, test_data.anchor_block)
current_time = state.slot * spec.config.SECONDS_PER_SLOT + store.genesis_time
on_tick_and_append_step(spec, store, current_time, test_steps)
assert store.time == current_time

# Apply generated blocks
for block_message in test_data.blocks:
block = block_message.payload.message
yield from tick_and_add_block(spec, store, block_message.payload, test_steps)
block_root = block.hash_tree_root()
assert store.blocks[block_root] == block

# Final steps (advance store to the current epoch)
apply_final_steps(spec, store, model_params, test_steps)
yield from yield_fork_choice_test_case(spec, store, test_data, debug)
# Run sanity checks against model params
run_sanity_checks(spec, store, model_params, target_block_root)

output_store_checks(spec, store, test_steps, with_filtered_block_weights=True)

yield 'steps', test_steps
71 changes: 2 additions & 69 deletions tests/generators/fork_choice_generated/instantiators/block_tree.py
Original file line number Diff line number Diff line change
@@ -6,13 +6,6 @@
from eth2spec.test.helpers.attester_slashings import (
get_valid_attester_slashing_by_indices,
)
from eth2spec.test.helpers.fork_choice import (
on_tick_and_append_step,
add_attestation,
add_attester_slashing,
add_block,
output_store_checks,
)
from eth2spec.test.helpers.state import (
transition_to,
next_slot,
@@ -27,11 +20,11 @@
advance_state_to_anchor_epoch,
produce_block,
attest_to_slot,
yield_fork_choice_test_case,
)
from .debug_helpers import (
attesters_in_block,
print_epoch,
print_head,
print_block_tree,
)

@@ -523,12 +516,6 @@ def split_list(lst, n):
sorted(out_of_block_attester_slashing_messages, key=lambda a: a.payload.attestation_1.data.slot))


def _on_tick_and_append_step(spec, store, slot, test_steps):
time = slot * spec.config.SECONDS_PER_SLOT + store.genesis_time
on_tick_and_append_step(spec, store, time, test_steps)
assert store.time == time


def gen_block_tree_test_data(spec,
state,
debug,
@@ -602,59 +589,5 @@ def yield_block_tree_test_case(spec,
test_data = gen_block_tree_test_data(spec, state, debug, seed, sm_links, block_parents,
with_attester_slashings, with_invalid_messages)

# Yield meta
for k, v in test_data.meta.items():
yield k, 'meta', v

# Yield anchor state and block initialization
yield 'anchor_state', test_data.anchor_state
yield 'anchor_block', test_data.anchor_block

test_steps = []
store = spec.get_forkchoice_store(test_data.anchor_state, test_data.anchor_block)
_on_tick_and_append_step(spec, store, test_data.anchor_state.slot, test_steps)

# Apply generated messages
max_block_slot = max(b.payload.message.slot for b in test_data.blocks)
max_attestation_slot = max(a.payload.data.slot for a in test_data.atts) if any(test_data.atts) else 0
max_slashing_slot = max(
s.payload.attestation_1.data.slot for s in test_data.slashings) if any(test_data.slashings) else 0

start_slot = min(b.payload.message.slot for b in test_data.blocks)
end_slot = max(max_block_slot, max_attestation_slot, max_slashing_slot) + 1

# Advance time to start_slot
_on_tick_and_append_step(spec, store, start_slot, test_steps)

# Apply messages to store
for slot in range(start_slot, end_slot + 1):
# on_tick
_on_tick_and_append_step(spec, store, slot, test_steps)

# on_attestation for attestations from the previous slot
for attestation_message in (a for a in test_data.atts if a.payload.data.slot == slot - 1):
yield from add_attestation(spec, store, attestation_message.payload,
test_steps, valid=attestation_message.valid)

# on_attester_slashing for slashing from the previous slot
for attester_slashing_message in (s for s in test_data.slashings
if s.payload.attestation_1.data.slot == slot - 1):
yield from add_attester_slashing(spec, store, attester_slashing_message.payload,
test_steps, valid=attester_slashing_message.valid)

# on_block for blocks from the current slot
for signed_block_message in (b for b in test_data.blocks if b.payload.message.slot == slot):
yield from add_block(spec, store, signed_block_message.payload, test_steps, signed_block_message.valid)

block_root = signed_block_message.payload.message.hash_tree_root()
if signed_block_message.valid:
assert store.blocks[block_root] == signed_block_message.payload.message
else:
assert block_root not in store.blocks.values()

if debug:
print(' head: ' + print_head(spec, store))

output_store_checks(spec, store, test_steps, with_filtered_block_weights=True)

yield 'steps', test_steps
yield from yield_fork_choice_test_case(spec, store, test_data, debug)
83 changes: 79 additions & 4 deletions tests/generators/fork_choice_generated/instantiators/helpers.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from dataclasses import dataclass
from dataclasses import dataclass, field
from .debug_helpers import print_epoch
from eth2spec.test.helpers.state import (
next_slot,
@@ -10,7 +10,14 @@
build_empty_block,
sign_block,
)

from eth2spec.test.helpers.fork_choice import (
on_tick_and_append_step,
add_attestation,
add_attester_slashing,
add_block,
output_store_checks,
)
from .debug_helpers import print_head

@dataclass
class ProtocolMessage:
@@ -24,8 +31,9 @@ class FCTestData:
anchor_block: object
anchor_state: object
blocks: list[ProtocolMessage]
atts: list[ProtocolMessage]
slashings: list[ProtocolMessage]
atts: list[ProtocolMessage] = field(default_factory=list)
slashings: list[ProtocolMessage] = field(default_factory=list)
store_final_time: int = 0


class BranchTip:
@@ -216,3 +224,70 @@ def advance_state_to_anchor_epoch(spec, state, anchor_epoch, debug) -> ([], Bran
return signed_blocks, anchor_tip


def _on_tick_and_append_step(spec, store, slot, test_steps):
time = slot * spec.config.SECONDS_PER_SLOT + store.genesis_time
on_tick_and_append_step(spec, store, time, test_steps)
assert store.time == time


def yield_fork_choice_test_case(spec, store, test_data: FCTestData, debug: bool):
# Yield meta
for k, v in test_data.meta.items():
yield k, 'meta', v

# Yield anchor state and block initialization
yield 'anchor_state', test_data.anchor_state
yield 'anchor_block', test_data.anchor_block

test_steps = []
_on_tick_and_append_step(spec, store, test_data.anchor_state.slot, test_steps)

# Apply generated messages
max_block_slot = max(b.payload.message.slot for b in test_data.blocks)
max_attestation_slot = max(a.payload.data.slot for a in test_data.atts) + 1 if any(test_data.atts) else 0
max_slashing_slot = max(max(s.payload.attestation_1.data.slot,
s.payload.attestation_2.data.slot)
for s in test_data.slashings) + 1 if any(test_data.slashings) else 0

start_slot = min(b.payload.message.slot for b in test_data.blocks)
end_slot = max(max_block_slot, max_attestation_slot, max_slashing_slot)

# Advance time to start_slot
_on_tick_and_append_step(spec, store, start_slot, test_steps)

# Apply messages to store
for slot in range(start_slot, end_slot + 1):
# on_tick
_on_tick_and_append_step(spec, store, slot, test_steps)

# on_attestation for attestations from the previous slot
for attestation_message in (a for a in test_data.atts if a.payload.data.slot == slot - 1):
yield from add_attestation(spec, store, attestation_message.payload,
test_steps, valid=attestation_message.valid)

# on_attester_slashing for slashing from the previous slot
for attester_slashing_message in (s for s in test_data.slashings
if max(s.payload.attestation_1.data.slot,
s.payload.attestation_2.data.slot) == slot - 1):
yield from add_attester_slashing(spec, store, attester_slashing_message.payload,
test_steps, valid=attester_slashing_message.valid)

# on_block for blocks from the current slot
for signed_block_message in (b for b in test_data.blocks if b.payload.message.slot == slot):
yield from add_block(spec, store, signed_block_message.payload, test_steps, signed_block_message.valid)

block_root = signed_block_message.payload.message.hash_tree_root()
if signed_block_message.valid:
assert store.blocks[block_root] == signed_block_message.payload.message
else:
assert block_root not in store.blocks.values()

if store.time < test_data.store_final_time:
on_tick_and_append_step(spec, store, test_data.store_final_time, test_steps)

if debug:
print(' head: ' + print_head(spec, store))

output_store_checks(spec, store, test_steps, with_filtered_block_weights=True)

yield 'steps', test_steps