Flaky or Flakey e2e tests

[piersy]

Two of the 7 e2e tests seem to fail randomly when running locally this commit 48d7851 (current tip of celo8)

Run test_fee_currency_fails_intrinsic.sh
+ tail -F -n 0 geth.log
+ sleep 0.2
++ deploy_fee_currency false false true
+++ forge create --root /Users/pierspowlesland/projects/op-geth/e2e_test/debug-fee-currency --contracts /Users/pierspowlesland/projects/op-geth/e2e_test/debug-fee-currency --private-key 0x2771aff413cac48d9f8c114fabddd9195a2129f3c2c436caa07e27bb7f58ead5 DebugFeeCurrency.sol:DebugFeeCurrency --constructor-args 100000000000000000000000000 false false true --json
+++ jq .deployedTo -r
++ local fee_currency=0xD5F78a597b68383c75A360d4Fbd4C86A2f8887e2
++ '[' -z 0xD5F78a597b68383c75A360d4Fbd4C86A2f8887e2 ']'
++ cast send --private-key 0x2771aff413cac48d9f8c114fabddd9195a2129f3c2c436caa07e27bb7f58ead5 0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb0003 'setExchangeRate(address, uint256, uint256)' 0xD5F78a597b68383c75A360d4Fbd4C86A2f8887e2 2ether 1ether
++ cast send --private-key 0x2771aff413cac48d9f8c114fabddd9195a2129f3c2c436caa07e27bb7f58ead5 0x9212Fb72ae65367A7c887eC4Ad9bE310BAC611BF 'setCurrencyConfig(address, address, uint256)' 0xD5F78a597b68383c75A360d4Fbd4C86A2f8887e2 0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb0003 60000
++ echo 0xD5F78a597b68383c75A360d4Fbd4C86A2f8887e2
+ fee_currency=0xD5F78a597b68383c75A360d4Fbd4C86A2f8887e2
+ cip_64_tx 0xD5F78a597b68383c75A360d4Fbd4C86A2f8887e2 1 true
+ assert_cip_64_tx false
+ local value
+ read -r value
++ cast chain-id
+ /Users/pierspowlesland/projects/op-geth/e2e_test/js-tests/send_tx.mjs 1337 0x2771aff413cac48d9f8c114fabddd9195a2129f3c2c436caa07e27bb7f58ead5 0xD5F78a597b68383c75A360d4Fbd4C86A2f8887e2 1 true
+ local expected_error=
++ echo '{"success":false,"replaced":false,"error":null}'
++ jq .success
+ '[' false '!=' false ']'
+ '[' -z '' ']'
+ expected_error=null
+ echo '{"success":false,"replaced":false,"error":null}'
+ jq .error
+ grep -qE null
+ sleep 2
+ cip_64_tx 0xD5F78a597b68383c75A360d4Fbd4C86A2f8887e2 1 true
+ assert_cip_64_tx false
+ local value
+ read -r value
++ cast chain-id
+ /Users/pierspowlesland/projects/op-geth/e2e_test/js-tests/send_tx.mjs 1337 0x2771aff413cac48d9f8c114fabddd9195a2129f3c2c436caa07e27bb7f58ead5 0xD5F78a597b68383c75A360d4Fbd4C86A2f8887e2 1 true
+ local expected_error=
++ echo '{"success":false,"replaced":false,"error":null}'
++ jq .success
+ '[' false '!=' false ']'
+ '[' -z '' ']'
+ expected_error=null
+ echo '{"success":false,"replaced":false,"error":null}'
+ jq .error
+ grep -qE null
+ cleanup_fee_currency 0xD5F78a597b68383c75A360d4Fbd4C86A2f8887e2
+ local fee_currency=0xD5F78a597b68383c75A360d4Fbd4C86A2f8887e2
+ cast send --private-key 0x2771aff413cac48d9f8c114fabddd9195a2129f3c2c436caa07e27bb7f58ead5 0x9212Fb72ae65367A7c887eC4Ad9bE310BAC611BF 'removeCurrencies(address, uint256)' 0xD5F78a597b68383c75A360d4Fbd4C86A2f8887e2 2 --json
+ jq 'if .status == "0x1" then 0 else 1 end' -r
0
+ sleep 0.5
+ kill %1
++ grep -Ec 'fee-currency EVM execution error, temporarily blocking fee-currency in local txpools .+ surpassed maximum allowed intrinsic gas for CreditFees\(\) in fee-currency' debug-fee-currency/geth.intrinsic.log
+ '[' 6 -ne 1 ']'
+ exit 1
FAIL test_fee_currency_fails_intrinsic.sh ❌

Run test_fee_currency_fails_on_credit.sh
+ tail -F -n0 geth.log
+ sleep 0.2
++ deploy_fee_currency false true false
+++ forge create --root /Users/pierspowlesland/projects/op-geth/e2e_test/debug-fee-currency --contracts /Users/pierspowlesland/projects/op-geth/e2e_test/debug-fee-currency --private-key 0x2771aff413cac48d9f8c114fabddd9195a2129f3c2c436caa07e27bb7f58ead5 DebugFeeCurrency.sol:DebugFeeCurrency --constructor-args 100000000000000000000000000 false true false --json
+++ jq .deployedTo -r
++ local fee_currency=0x3D324b41CDf6654F85d16F9abB79775F0B8F510A
++ '[' -z 0x3D324b41CDf6654F85d16F9abB79775F0B8F510A ']'
++ cast send --private-key 0x2771aff413cac48d9f8c114fabddd9195a2129f3c2c436caa07e27bb7f58ead5 0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb0003 'setExchangeRate(address, uint256, uint256)' 0x3D324b41CDf6654F85d16F9abB79775F0B8F510A 2ether 1ether
++ cast send --private-key 0x2771aff413cac48d9f8c114fabddd9195a2129f3c2c436caa07e27bb7f58ead5 0x9212Fb72ae65367A7c887eC4Ad9bE310BAC611BF 'setCurrencyConfig(address, address, uint256)' 0x3D324b41CDf6654F85d16F9abB79775F0B8F510A 0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb0003 60000
++ echo 0x3D324b41CDf6654F85d16F9abB79775F0B8F510A
+ fee_currency=0x3D324b41CDf6654F85d16F9abB79775F0B8F510A
+ cip_64_tx 0x3D324b41CDf6654F85d16F9abB79775F0B8F510A 1 true
+ assert_cip_64_tx false
+ local value
+ read -r value
++ cast chain-id
+ /Users/pierspowlesland/projects/op-geth/e2e_test/js-tests/send_tx.mjs 1337 0x2771aff413cac48d9f8c114fabddd9195a2129f3c2c436caa07e27bb7f58ead5 0x3D324b41CDf6654F85d16F9abB79775F0B8F510A 1 true
+ local expected_error=
++ echo '{"success":false,"replaced":false,"error":null}'
++ jq .success
+ '[' false '!=' false ']'
+ '[' -z '' ']'
+ expected_error=null
+ echo '{"success":false,"replaced":false,"error":null}'
+ jq .error
+ grep -qE null
+ sleep 2
+ cip_64_tx 0x3D324b41CDf6654F85d16F9abB79775F0B8F510A 1 true
+ assert_cip_64_tx false
+ local value
+ read -r value
++ cast chain-id
+ /Users/pierspowlesland/projects/op-geth/e2e_test/js-tests/send_tx.mjs 1337 0x2771aff413cac48d9f8c114fabddd9195a2129f3c2c436caa07e27bb7f58ead5 0x3D324b41CDf6654F85d16F9abB79775F0B8F510A 1 true
+ local expected_error=
++ echo '{"success":false,"replaced":false,"error":null}'
++ jq .success
+ '[' false '!=' false ']'
+ '[' -z '' ']'
+ expected_error=null
+ echo '{"success":false,"replaced":false,"error":null}'
+ jq .error
+ grep -qE null
+ cleanup_fee_currency 0x3D324b41CDf6654F85d16F9abB79775F0B8F510A
+ local fee_currency=0x3D324b41CDf6654F85d16F9abB79775F0B8F510A
+ cast send --private-key 0x2771aff413cac48d9f8c114fabddd9195a2129f3c2c436caa07e27bb7f58ead5 0x9212Fb72ae65367A7c887eC4Ad9bE310BAC611BF 'removeCurrencies(address, uint256)' 0x3D324b41CDf6654F85d16F9abB79775F0B8F510A 2 --json
+ jq 'if .status == "0x1" then 0 else 1 end' -r
0
+ sleep 0.5
+ kill %1
+ grep '' debug-fee-currency/geth.partial.log
Binary file debug-fee-currency/geth.partial.log matches
./test_fee_currency_fails_on_credit.sh: line 32: 21632 Terminated: 15          tail -F -n0 geth.log > debug-fee-currency/geth.partial.log
++ grep -Ec 'fee-currency EVM execution error, temporarily blocking fee-currency in local txpools .+ This DebugFeeCurrency always fails in \(old\) creditGasFees!' debug-fee-currency/geth.partial.log
+ '[' 2 -ne 1 ']'
+ exit 1
FAIL test_fee_currency_fails_on_credit.sh ❌

[ezdac]

It could be that this is buggy behavior by the node,
but I am mainly suspecting this to be an issue with cleanup of the logfiles that are piped to during the tests and then grep`ed for certain log strings only appearing once.

The failure stems from them appearing more than once (+ '[' 6 -ne 1 ']' and + '[' 2 -ne 1 ']')

I had this locally before, and deleting the debug-fee-currency/geth.intrinsic.log and debug-fee-currency/geth.partial.log files before running the tests always helped - so this is more likely a cleanup issue.