Add storage bounds for pallet `staking` and clean up deprecated non paged exposure storages #6445

re-gius · 2024-11-11T15:08:49Z

This is part of #6289 and necessary for the Asset Hub migration.

Building on the observations and suggestions from #255 .

Changes

Add MaxInvulnerables to bound Invulnerables Vec -> BoundedVec.
- Set to constant 20 in the pallet (must be >= 17 for backward compatibility of runtime westend).
Add MaxDisabledValidators to bound DisabledValidators Vec -> BoundedVec
- Set to constant 100 in the pallet (it should be <= 1/3 * MaxValidatorsCount according to the current disabling strategy)
Remove ErasStakers and ErasStakersClipped (see Tracker issue for cleaning up old non-paged exposure logic in staking pallet #433 )
- They were deprecated in v14 and could have been removed since staking era 1504 (now it's > 1600)
- Completing the task from [pallet-staking] Clean up stale non paged exposure storages in the pallet #5986
Use MaxExposurePageSize to bound ErasStakersPaged mapping to exposure pages: each ExposurePage.others Vec is turned into a WeakBoundedVec to allow easy and quick changes to this bound
Add MaxBondedEras to bound BondedEras Vec -> BoundedVec
- Set to BondingDuration::get() + 1 everywhere to include both time interval endpoints in [current_era - BondingDuration::get(), current_era]. Notice that this was done manually in every test and runtime, so I wonder if there is a better way to ensure that MaxBondedEras::get() == BondingDuration::get() + 1 everywhere.
Add MaxRewardPagesPerValidator to bound ClaimedRewards Vec of pages -> WeakBoundedVec
- Set to constant 20 in the pallet. The vector of pages is now a WeakBoundedVec to allow easy and quick changes to this parameter
Remove MaxValidatorsCount optional storage item to add MaxValidatorsCount mandatory config parameter
- Using it to to bound EraRewardPoints.individual BTreeMap -> BoundedBTreeMap;
- Set to dynamic parameter in runtime westend so that changing it should not require migrations for it

TO DO
Slashing storage items will be bounded in another PR.

UnappliedSlashes
SlashingSpans

Co-authored-by: Kian Paimani <[email protected]>

substrate/frame/staking/src/pallet/mod.rs

substrate/primitives/staking/src/lib.rs

seadanda · 2025-01-16T16:50:41Z

polkadot/runtime/test-runtime/src/lib.rs

@@ -400,7 +401,7 @@ impl pallet_staking::Config for Runtime {
 	type DisablingStrategy = pallet_staking::UpToLimitWithReEnablingDisablingStrategy;
 	type MaxInvulnerables = ConstU32<20>;
 	type MaxRewardPagesPerValidator = ConstU32<20>;
-	type MaxValidatorsCount = ConstU32<300>;
+	type MaxValidatorsCount = MaxAuthorities;


This is a huge jump, is there a reason why the MaxAuthorities is so big in the test runtime?

Oh, right! Then it's probably better to revert the change in that case, adding also a comment: 6c9806c

gui1117

Overall looks good, but there is still some comments to resolve.

gui1117 · 2025-01-17T05:31:31Z

substrate/frame/staking/src/lib.rs

+	pub fn from_clipped(exposure: Exposure<AccountId, Balance>) -> Result<Self, ()> {
+		let old_exposures = exposure.others.len();
+		let others = WeakBoundedVec::try_from(exposure.others).unwrap_or_default();
+		defensive_assert!(old_exposures == others.len(), "Too many exposures for a page");


This function is not used, we can make it better or remove it.

Suggested change

pub fn from_clipped(exposure: Exposure<AccountId, Balance>) -> Result<Self, ()> {

let old_exposures = exposure.others.len();

let others = WeakBoundedVec::try_from(exposure.others).unwrap_or_default();

defensive_assert!(old_exposures == others.len(), "Too many exposures for a page");

pub fn try_from_clipped(exposure: Exposure<AccountId, Balance>) -> Result<Self, ()> {

let others = WeakBoundedVec::try_from(exposure.others).map_err(|_| ())?;

Changed in 3b7bc95

substrate/frame/staking/src/lib.rs

gui1117 · 2025-01-17T05:39:36Z

substrate/frame/staking/src/lib.rs

-		claimed_pages.push(page);
+		// try to add page to claimed entries
+		if claimed_pages.try_push(page).is_err() {
+			defensive!("Limit reached for maximum number of pages.");


The proof should be more precise. In what circumstance can this limit be reached, why is it impossible in practice.

What do you mean by "the proof"?

gui1117 · 2025-01-17T05:54:16Z

substrate/frame/staking/src/migrations.rs

+			let mut eras_stakers_keys =
+				v16::ErasStakers::<T>::iter_keys().map(|(k1, _k2)| k1).collect::<Vec<_>>();
+			eras_stakers_keys.dedup();
+			for k in eras_stakers_keys {
+				let mut removal_result =
+					v16::ErasStakers::<T>::clear_prefix(k, u32::max_value(), None);
+				while let Some(next_cursor) = removal_result.maybe_cursor {
+					removal_result = v16::ErasStakers::<T>::clear_prefix(
+						k,
+						u32::max_value(),
+						Some(&next_cursor[..]),
+					);
+				}
+			}


This seems to try to remove all keys in one time, if we don't need multi-block migration and we are sure it is ok then we can do:

Suggested change

let mut eras_stakers_keys =

v16::ErasStakers::<T>::iter_keys().map(|(k1, _k2)| k1).collect::<Vec<_>>();

eras_stakers_keys.dedup();

for k in eras_stakers_keys {

let mut removal_result =

v16::ErasStakers::<T>::clear_prefix(k, u32::max_value(), None);

while let Some(next_cursor) = removal_result.maybe_cursor {

removal_result = v16::ErasStakers::<T>::clear_prefix(

k,

u32::max_value(),

Some(&next_cursor[..]),

);

}

}

v16::ErasStakers::<T>::clear(u32::max_value(), None);

We can be sure for Polkadot or Kusama but how can we be sure for every other chain? Maybe there are some limits I'm not aware of

gui1117 · 2025-01-17T05:54:48Z

substrate/frame/staking/src/migrations.rs

+			let mut eras_stakers_clipped_keys = v16::ErasStakersClipped::<T>::iter_keys()
+				.map(|(k1, _k2)| k1)
+				.collect::<Vec<_>>();
+			eras_stakers_clipped_keys.dedup();
+			for k in eras_stakers_clipped_keys {
+				let mut removal_result =
+					v16::ErasStakersClipped::<T>::clear_prefix(k, u32::max_value(), None);
+				while let Some(next_cursor) = removal_result.maybe_cursor {
+					removal_result = v16::ErasStakersClipped::<T>::clear_prefix(
+						k,
+						u32::max_value(),
+						Some(&next_cursor[..]),
+					);
+				}
+			}


Suggested change

let mut eras_stakers_clipped_keys = v16::ErasStakersClipped::<T>::iter_keys()

.map(|(k1, _k2)| k1)

.collect::<Vec<_>>();

eras_stakers_clipped_keys.dedup();

for k in eras_stakers_clipped_keys {

let mut removal_result =

v16::ErasStakersClipped::<T>::clear_prefix(k, u32::max_value(), None);

while let Some(next_cursor) = removal_result.maybe_cursor {

removal_result = v16::ErasStakersClipped::<T>::clear_prefix(

k,

u32::max_value(),

Some(&next_cursor[..]),

);

}

}

v16::ErasStakersClipped::<T>::clear(u32::max_value(), None);

Same here, are we assuming that this is safe?

gui1117 · 2025-01-17T05:58:20Z

substrate/frame/staking/src/migrations.rs

+			} else {
+				log!(info, "v17 applied successfully.");
+			}
+			T::DbWeight::get().reads_writes(1, 1)


we can count the number of operation as we do them.

How can we do that?

Co-authored-by: Guillaume Thiolliere <[email protected]>

paritytech-workflow-stopper · 2025-01-21T17:04:18Z

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/12891905705
Failed job name: fmt

add MaxInvulnerables

6c808b6

re-gius added the T2-pallets This PR/Issue is related to a particular pallet. label Nov 11, 2024

re-gius self-assigned this Nov 11, 2024

This was referenced Nov 11, 2024

[pallet-staking] Bound all storage #5987

Closed

Tracking issue for bounding pallet-staking storage items #255

Open

[tracking] remove without_storage_info for pallets migrating to AH #6289

Open

re-gius added 23 commits November 13, 2024 14:51

add MaxActiveValidators

1b18bb1

Merge branch 'master' into re-gius/staking/add-storage-bounds

0ac60af

fix + start working on migrations

4edb8e1

nit (typo)

65c7de6

fix migrations for BoundedVec

35069c9

Merge branch 'master' into re-gius/staking/add-storage-bounds

d4b15af

Migrate ErasRewardPoints

eb93a70

small fixes - BoundedBTreeMap

58cf45d

Merge branch 'master' into re-gius/staking/add-storage-bounds

292c5d0

fix

74f2055

Merge branch 'master' into re-gius/staking/add-storage-bounds

63f2c5e

Merge branch 'master' into re-gius/staking/add-storage-bounds

b249a67

Adapt to new recent DisabledValidators change

362f505

Change MaxActiveValidators to MaxWinners

4f21d53

undo change from another PR

e258baf

Merge branch 'master' into re-gius/staking/add-storage-bounds

4b7d896

small fix for MaxWinners

01e2233

small fixes

243bec8

Merge branch 'master' into re-gius/staking/add-storage-bounds

a599afc

Remove deprecated ErasStakers and ErasStakersClipped

9a25fbc

Fixes from #5986

98daabc

nit

aec5087

remove unused import

36204da

paritytech deleted a comment from paritytech-workflow-stopper bot Nov 25, 2024

re-gius added 5 commits December 19, 2024 16:12

nit

c889905

migration fix

300b7f9

fix migrations

fc63dc7

Merge branch 'master' into re-gius/staking/add-storage-bounds

720c9eb

Set MaxValidatorsCount equal to MaxAuthorities

a065a91

re-gius mentioned this pull request Dec 20, 2024

Unify bounds for the maximum number of validators across pallets #6972

Open

re-gius and others added 6 commits December 20, 2024 13:19

Update polkadot/runtime/test-runtime/src/lib.rs

483eb03

Co-authored-by: Kian Paimani <[email protected]>

Merge branch 'master' into re-gius/staking/add-storage-bounds

7aeb8a5

fix comment

c51d09b

fix

1a9be6b

Update pr_6445.prdoc from semver CI

235a260

Merge branch 'master' into re-gius/staking/add-storage-bounds

83a888b

paritytech deleted a comment from paritytech-workflow-stopper bot Jan 2, 2025

Merge branch 'master' into re-gius/staking/add-storage-bounds

b2efa18

gui1117 reviewed Jan 16, 2025

View reviewed changes

substrate/frame/staking/src/pallet/mod.rs Show resolved Hide resolved

substrate/frame/staking/src/pallet/mod.rs Outdated Show resolved Hide resolved

substrate/primitives/staking/src/lib.rs Show resolved Hide resolved

seadanda reviewed Jan 16, 2025

View reviewed changes

gui1117 reviewed Jan 17, 2025

View reviewed changes

Merge branch 'master' into re-gius/staking/add-storage-bounds

78539f0

paritytech-review-bot bot requested a review from a team January 21, 2025 16:22

re-gius and others added 8 commits January 21, 2025 17:29

add more info on deleted storage items to prdoc

1a39de4

add error when processing non-paged exposures

3b7bc95

Update substrate/frame/staking/src/pallet/mod.rs

dc93f0f

Co-authored-by: Guillaume Thiolliere <[email protected]>

Update substrate/primitives/staking/src/lib.rs

e6c9942

Co-authored-by: Guillaume Thiolliere <[email protected]>

Update substrate/frame/staking/src/lib.rs

d91414b

Co-authored-by: Guillaume Thiolliere <[email protected]>

Update substrate/frame/staking/src/lib.rs

9d2e835

Co-authored-by: Guillaume Thiolliere <[email protected]>

Update substrate/frame/staking/src/lib.rs

88d20a3

Co-authored-by: Guillaume Thiolliere <[email protected]>

fmt

649b0f9

re-gius added 2 commits January 21, 2025 18:07

Change MaxValidatorsCount in test runtime

6c9806c

Merge branch 'master' into re-gius/staking/add-storage-bounds

6ca4347

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add storage bounds for pallet `staking` and clean up deprecated non paged exposure storages #6445

Add storage bounds for pallet `staking` and clean up deprecated non paged exposure storages #6445

re-gius commented Nov 11, 2024 •

edited

Loading

seadanda Jan 16, 2025

re-gius Jan 21, 2025

gui1117 left a comment

gui1117 Jan 17, 2025

re-gius Jan 21, 2025

gui1117 Jan 17, 2025

re-gius Jan 21, 2025

gui1117 Jan 17, 2025

re-gius Jan 21, 2025

gui1117 Jan 17, 2025

re-gius Jan 21, 2025

gui1117 Jan 17, 2025

re-gius Jan 21, 2025

paritytech-workflow-stopper bot commented Jan 21, 2025

Add storage bounds for pallet staking and clean up deprecated non paged exposure storages #6445

Are you sure you want to change the base?

Add storage bounds for pallet staking and clean up deprecated non paged exposure storages #6445

Conversation

re-gius commented Nov 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gui1117 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paritytech-workflow-stopper bot commented Jan 21, 2025

Add storage bounds for pallet `staking` and clean up deprecated non paged exposure storages #6445

Add storage bounds for pallet `staking` and clean up deprecated non paged exposure storages #6445

re-gius commented Nov 11, 2024 •

edited

Loading