Resource Isolation Feature: Extend CACHE SELECT to allow warming up backup replicas

[anatoly2]

Why I'm doing:

Currently a cache select statement meant to bring in 2 replicas of each tablet in some resource isolation groups might look like CACHE SELECT * FROM table PROPERTIES( "resource_isolation_groups"="writegroup, async", "num_replicas"="2");
This would intend to bring in each tablet into 2 CN in writegroup, and 2 CN in async. However, there may be some situation where we just want to bring in data for fallback purposes. For example, say we have resource isolation "writegroup" which is responsible for data ingestion. As data is being ingested, it is automatically cached in the CN which is primarily responsible for each tablet. We don't need to do cache warmup for the primary replica because it's already cached, and it costs CPU to perform this noop. Instead, we'd like to issue a cache warmup such that we bring in backup replicas.

What I'm doing:

I added the following syntax to support backup replicas:
CACHE SELECT * FROM table PROPERTIES( "resource_isolation_groups"="writegroup", "num_backup_replicas"="1");
So this command will warmup the first backup replica. We could also specify num_backup_replicas=2 if we wanted 3 replicas total, etc.

https://jira.pinadmin.com/browse/RTA-6821

What type of PR is this:

• BugFix
• Feature
• Enhancement
• Refactor
• UT
• Doc
• Tool

Does this PR entail a change in behavior?

• Yes, this PR will result in a change in behavior.
• No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• Interface/UI changes: syntax, type conversion, expression evaluation, display information
• Parameter changes: default values, similar parameters but with different default values
• Policy changes: use new policy to replace old one, functionality automatically enabled
• Feature removed
• Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• I have added test cases for my bug fix or my new feature
• This pr needs user documentation (for new or modified features or behaviors)
• I have added documentation for my new feature or new function
• This is a backport pr

Bugfix cherry-pick branch check:

• I have checked the version labels which the pr will be auto-backported to the target branch
  • 3.4
  • 3.3
  • 3.2
  • 3.1
  • 3.0

[ctbrennan]

Looks pretty good so far, left some comments.

[saahilbarai]

Question for my understanding -

Suppose we have a write group of CN nodes. Like you mentioned in the PR description, that write group would have already had a primary replica cached upon ingestion. If we wanted to have two replicas without this PR we would have to perform a check against the hash ring to verify that for a given tablet id there was already one replica cached and that check is what we want to get rid of by exposing a secondary parameter "BackupReplicas". Essentially we now delegate more granular control of caching to the system admin to remove that check against the hash ring.

If my understanding above is correct, how do I as a system admin know what combination of "DesiredReplicas" and "BackupReplicas" to issue when issuing a cache select statement. Is it mostly as simple as if the resource isolation group accepts writes then I exclusively modify "BackupReplicas"?

[ctbrennan]

Suppose we have a write group of CN nodes. Like you mentioned in the PR description, that write group would have already had a primary replica cached upon ingestion. If we wanted to have two replicas without this PR we would have to perform a check against the hash ring to verify that for a given tablet id there was already one replica cached and that check is what we want to get rid of by exposing a secondary parameter "BackupReplicas".

Close, but the hash ring itself can't actually expose whether a given key (tablet, in this case) has been mapped before. Second, we could use tabletMappingCount to see if we've requested some CN for a given tablet, but the lifecycle of the tabletMappingCount doesn't match the lifecycle of the data on the CN disk, so we can't actually rely on a tablet's presence in tabletMappingCount to tell us if the primary replica is cached. It might just be that a huge chunk of data has never been read by a given FE, and therefore the tabletMappingCount map might be basically empty, for example, and in this case we don't want to unnecessarily issue a huge cache warmup statement to all the CN.

Essentially we now delegate more granular control of caching to the system admin to remove that check against the hash ring.

Yes, this is exactly right.

Is it mostly as simple as if the resource isolation group accepts writes then I exclusively modify "BackupReplicas"?

Yes, this is exactly what I expect to do.

[ctbrennan]

LGTM, thanks Anatolijs. Could we make a private build and deploy in a dev cluster and run your test suite before we merge this in?

[zhenxiao]

looks good
I would suggest not having the num_backup_replicas property, but having a boolean isPrimary property.
If set isPrimary as true, the primary replica will be warmed up
if not, the primary replica will not be warmed up
Either way is fine. Just my 2 cents