Add WHISK fork only tests #3442

dapplion · 2023-07-05T15:31:23Z

Description

Adds a baseline of tests covering the whisk feature introduced with #3342. In summary:

generate and validate tracker, opening and shuffle proofs
error cases for each assert condition in the spec

Scope / future work

Tests are only run for the whisk fork and minimal. The current crypto backend is significantly slower than BLS and KZG, so running more tests than that is prohibitively expensive at the moment.

Also to run tests from previous forks on Whisk, the vector generation has to be modified to accommodate the fact that the proposer is not known, and requires validator_size factorial G1 multiplications in an implementation without revealed shuffling data to figure out who is the proposer for a whisk round.

All those points can be addressed, but in a future PR if necessary

tests/core/pyspec/eth2spec/test/context.py

dapplion · 2023-07-20T14:46:39Z

Note:

Bumped curdleproofs lib to use faster backend Use py_arkworks_bls12381 nalinbhardwaj/curdleproofs.pie#19

dapplion · 2023-07-24T18:16:48Z

@hwwhww seems circle CI is experiencing a caching problem again https://app.circleci.com/pipelines/github/ethereum/consensus-specs/9434/workflows/86a7977e-ac75-42f6-8d2e-a81982c949e4/jobs/70233

ImportError while loading conftest '/home/circleci/specs-repo/tests/core/pyspec/eth2spec/test/conftest.py'.
eth2spec/test/conftest.py:1: in <module>
    from eth2spec.test import context
eth2spec/test/context.py:12: in <module>
    from eth2spec.whisk import mainnet as spec_whisk_mainnet, minimal as spec_whisk_minimal
eth2spec/whisk/__init__.py:1: in <module>
    from . import mainnet as spec  # noqa:F401
eth2spec/whisk/mainnet.py:222: in <module>
    CURDLEPROOFS_CRS = curdleproofs.CurdleproofsCrs.from_random_points(                 WHISK_VALIDATORS_PER_SHUFFLE,                 CURDLEPROOFS_N_BLINDERS,                 [G1Point.from_compressed_bytes_unchecked(p) for p in CURDLEPROOFS_CRS_G1]             )  # noqa: E501
E   AttributeError: type object 'CurdleproofsCrs' has no attribute 'from_random_points'

this method from_random_points exists in the referenced version and github CI and my local env pass tests

hwwhww · 2023-07-27T06:21:56Z

@dapplion

Oh my, it's so weird. I pushed a commit in my branch: ccd8026, and it seems to have installed with the correct commit but has some other errors.

But after I pushed that commit to this PR's branch, it showed the same error.

Is it possible because your repository has a different CircleCI setting?

dapplion · 2023-07-28T17:57:37Z

Is it possible because your repository has a different CircleCI setting?

The reported errors were correct, there was a type issue with curdleproof's return values not caught in my local env due to caching.

With nalinbhardwaj/curdleproofs.pie#21 merged, now all tests should pass :)

EDIT: Circle CI version cache issue happening again. It's strange since the version bump is from a version that has from_random_points to a newer version that also has said method

ImportError while loading conftest '/home/circleci/specs-repo/tests/core/pyspec/eth2spec/test/conftest.py'.
eth2spec/test/conftest.py:1: in <module>
    from eth2spec.test import context
eth2spec/test/context.py:12: in <module>
    from eth2spec.whisk import mainnet as spec_whisk_mainnet, minimal as spec_whisk_minimal
eth2spec/whisk/__init__.py:1: in <module>
    from . import mainnet as spec  # noqa:F401
eth2spec/whisk/mainnet.py:222: in <module>
    CURDLEPROOFS_CRS = curdleproofs.CurdleproofsCrs.from_random_points(WHISK_VALIDATORS_PER_SHUFFLE, CURDLEPROOFS_N_BLINDERS, [G1Point.from_compressed_bytes_unchecked(p) for p in CURDLEPROOFS_CRS_G1])  # noqa: E501
E   AttributeError: type object 'CurdleproofsCrs' has no attribute 'from_random_points'

dapplion · 2023-08-17T11:56:46Z

After debugging on a separate branch https://github.com/ethereum/consensus-specs/tree/whisk-tests-circleci results show that circle CI instances declare to install the correct commit of curdleproofs, but then a very old commit is actually used. I have not understood where this old version is coming from, but forcing an uninstall + reinstall seems to allow downstream steps to use the right version. I would suggest re-considering CircleCI as a platform to test the specs due to its non-determinism.

Note this hack forces mantainers to duplicate the version declaration in both a Makefile command (for CircleCI) and regular requirements (for Github Actions)

consensus-specs/Makefile

Lines 113 to 120 in db502dd

    
           install_test: preinstallation 
        
           	. venv/bin/activate; \ 
        
           	python3 -m pip install -e .; \ 
        
           	python3 -m pip install -e .[lint,test]; \ 
        
           	python3 -m pip freeze; \ 
        
           	python3 -m pip uninstall -y curdleproofs; \ 
        
           	python3 -m pip install "git+https://github.com/nalinbhardwaj/curdleproofs.pie@bc4bb34961a896af6bcf4a5ff4867bb2971742c8#egg=curdleproofs&subdirectory=curdleproofs"; \ 
        
           	python3 -m pip freeze;

dapplion · 2023-08-18T13:19:51Z

@hwwhww Published curdleproofs 🎉 this PR is no longer using a Github ref

consensus-specs/setup.py

Line 537 in a74b4fc

"curdleproofs==0.1.1",

specs/_features/whisk/beacon-chain.md

dapplion · 2023-10-10T11:21:15Z

Updated the CRS with correct serialization. Tests should pass now.

@hwwhww There's this error on CI which I can't reproduce locally. The error points to a spec variable not being present, however it's defined in the preset file. Any ideas?

 eth2spec/test/whisk/block_processing/test_process_shuffled_trackers.py:132: in test_invalid_gap_non_zero_proof
    set_state_epoch_selection_gap(spec, state)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

spec = <module 'eth2spec.whisk.minimal' from '/home/devops/consensus-specs-02/consensus-specs-02/consensus-specs/consensus-specs/tests/core/pyspec/eth2spec/whisk/minimal.py'>
state = BeaconState(Container)
    genesis_time: uint64 = 0
    genesis_validators_root: Root = 0x0a08c27fe4ece2483f9e581f78c6...e002bbe9af91c6de91ab693f6477551e7ee0a1e675d0fc614814d8a8aa
                                                           )

    def set_state_epoch_selection_gap(spec, state):
>       set_state_epoch(spec, state, spec.WHISK_EPOCHS_PER_SHUFFLING_PHASE - 1)
E       AttributeError: module 'eth2spec.whisk.minimal' has no attribute 'WHISK_EPOCHS_PER_SHUFFLING_PHASE'

Ref: https://github.com/ethereum/consensus-specs/actions/runs/6468128943/job/17559568960#step:8:106

mkalinin

The PR looks good to me!
@hwwhww what do you think about merging it after fixing the CI issue?

hwwhww

Hi @dapplion

I believe it was because these parameters are defined in Configurations, so we use spec.config.* to access them.

tests/core/pyspec/eth2spec/test/whisk/block_processing/test_process_shuffled_trackers.py

hwwhww

LGTM!

hwwhww · 2023-12-08T02:17:04Z

tests/core/pyspec/eth2spec/test/helpers/keys.py

+# Must be unique among the set `whisk_ks_initial + whisk_ks_final`
+def whisk_ks_final(i: int):
+    return i + 10000000


hwwhww · 2023-12-08T02:17:15Z

tests/core/pyspec/eth2spec/test/helpers/keys.py

+def whisk_ks_initial(i: int):
+    return i


perhaps add a TODO mark here

dapplion added 9 commits July 5, 2023 17:18

Add whisk tests

4ce2b02

Run on any push

829a59b

Fix test lint issues

91ff707

Update specs with state.whisk_trackers

7f9a026

FIx tests with rebase

cf975c6

Cache initial trackers

c2314dc

Add test_config_invariants

d6dcce0

Remove whisk from ALL_PHASES

758022d

Apply modularized setup changes

2ef22cb

dapplion force-pushed the whisk-tests branch from bf5fc29 to 2ef22cb Compare July 5, 2023 15:33

dapplion mentioned this pull request Jul 6, 2023

Whisk tests dapplion/consensus-specs#1

Open

hwwhww mentioned this pull request Jul 6, 2023

Upgrade wheel, pip, and setuptools with make install_test #3443

Merged

hwwhww added 3 commits July 7, 2023 01:40

Upgrade wheel pip setuptools with make install_test

8c65b21

Merge branch 'dev' into pr3442

71c25f8

kick the CI cache

5c194bb

hwwhww reviewed Jul 14, 2023

View reviewed changes

tests/core/pyspec/eth2spec/test/context.py Outdated Show resolved Hide resolved

dapplion added 2 commits July 20, 2023 16:44

Fix with_all_phases_from

6223491

Bump curdleproofs to use py_arkworks_bls12381

c0c453d

Update curdleproofs usage

211376e

dapplion force-pushed the whisk-tests branch from f87cf3e to 211376e Compare July 24, 2023 18:15

wip

ccd8026

Bump curdleproofs to fix upstream types

07ab004

force install correct version of curdleproofs

db502dd

dapplion force-pushed the whisk-tests branch from 0507b27 to 0c6e054 Compare August 18, 2023 13:19

Use published curdleproofs version

a74b4fc

dapplion force-pushed the whisk-tests branch from 0c6e054 to a74b4fc Compare August 18, 2023 15:52

mkalinin reviewed Aug 24, 2023

View reviewed changes

specs/_features/whisk/beacon-chain.md Outdated Show resolved Hide resolved

mkalinin reviewed Aug 24, 2023

View reviewed changes

specs/_features/whisk/beacon-chain.md Outdated Show resolved Hide resolved

hwwhww mentioned this pull request Sep 6, 2023

Pin curdleproofs==0.1.1 #3495

Merged

dapplion added 3 commits September 12, 2023 10:23

Add CRS files

4eca8e4

update CRS

de30a92

clean-up todo

5bab432

dapplion force-pushed the whisk-tests branch from 1788fab to 5bab432 Compare October 10, 2023 11:20

dapplion mentioned this pull request Oct 10, 2023

Generate CRS asn-d6/curdleproofs#17

Merged

mkalinin approved these changes Oct 12, 2023

View reviewed changes

hwwhww reviewed Oct 12, 2023

View reviewed changes

tests/core/pyspec/eth2spec/test/whisk/block_processing/test_process_shuffled_trackers.py Outdated Show resolved Hide resolved

tests/core/pyspec/eth2spec/test/whisk/block_processing/test_process_shuffled_trackers.py Outdated Show resolved Hide resolved

hwwhww and others added 2 commits October 16, 2023 17:22

Merge branch 'dev' into pr3442

d74932b

reference vars from config

f14900f

hwwhww added the scope:ssle label Oct 21, 2023

fix tests for mainnet preset

42d91d5

hwwhww approved these changes Dec 8, 2023

View reviewed changes

hwwhww merged commit bf09b9a into ethereum:dev Dec 8, 2023
28 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add WHISK fork only tests #3442

Add WHISK fork only tests #3442

dapplion commented Jul 5, 2023

dapplion commented Jul 20, 2023

dapplion commented Jul 24, 2023

hwwhww commented Jul 27, 2023 •

edited

Loading

dapplion commented Jul 28, 2023 •

edited

Loading

dapplion commented Aug 17, 2023

dapplion commented Aug 18, 2023 •

edited

Loading

dapplion commented Oct 10, 2023 •

edited

Loading

mkalinin left a comment

hwwhww left a comment

hwwhww left a comment

hwwhww Dec 8, 2023

hwwhww Dec 8, 2023

Add WHISK fork only tests #3442

Add WHISK fork only tests #3442

Conversation

dapplion commented Jul 5, 2023

Description

Scope / future work

dapplion commented Jul 20, 2023

dapplion commented Jul 24, 2023

hwwhww commented Jul 27, 2023 • edited Loading

dapplion commented Jul 28, 2023 • edited Loading

dapplion commented Aug 17, 2023

dapplion commented Aug 18, 2023 • edited Loading

dapplion commented Oct 10, 2023 • edited Loading

mkalinin left a comment

Choose a reason for hiding this comment

hwwhww left a comment

Choose a reason for hiding this comment

hwwhww left a comment

Choose a reason for hiding this comment

hwwhww Dec 8, 2023

Choose a reason for hiding this comment

hwwhww Dec 8, 2023

Choose a reason for hiding this comment

hwwhww commented Jul 27, 2023 •

edited

Loading

dapplion commented Jul 28, 2023 •

edited

Loading

dapplion commented Aug 18, 2023 •

edited

Loading

dapplion commented Oct 10, 2023 •

edited

Loading