Re-design datastore key queries ( JSON RPC API / GRPC)

[Leo-Besancon]

The JsonRPC API endpoint get_addresses lists, in the response, all the datastore keys of the given addresses.
One issue with this is that there is no limits in the number of datastore entries for a given address. So if the address has a lot of keys, the response may be too large (e.g. larger than the default 50Mo response limit of the API) which causes issues.

For example, the following explorer page errors out because of it: https://explorer.massa.net/mainnet/address/AS1Ba1T2mMpHvLEhTvsNkSDqm4djno2ezVojGrtLf1wRd8ThfZD3
A request / response to the mainnet API showing the issue:

{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "get_addresses",
    "params": [["AS1Ba1T2mMpHvLEhTvsNkSDqm4djno2ezVojGrtLf1wRd8ThfZD3"]]
}

{
    "jsonrpc": "2.0",
    "error": {
        "code": -32008,
        "message": "Response is too big",
        "data": "Exceeded max limit of 52428800"
    },
    "id": 1
}

TODO:

• List all the current ways we list datastore key (json rpc API, GRPC)
• Design a unified way to list the keys
  • Use (Bound, Bound) to define the range of keys to query. This is a way to generalize the optional prefix in some of the listing cases, with Bounds being Included, Excluded or Unbounded.
  • Limit the number of keys returned (max num would be a constant, or maybe a config parameter instead when applicable)
  • Best if non-breaking
• Implement and test all the key listing cases

[SlnPons]

Note priority N°1 is to address API JSONRPC+GRPC.

[damip]

How it would work:

• optional key_start parameter that can be either
  • absent (no lower bound)
  • INCLUDED(key)
  • EXCLUDED(key)
• same for optional key_end parameter:
  • absent (no upper bound)
  • INCLUDED(key)
  • EXCLUDED(key)
• optional LIMIT parameter (default 100) and error if the user tries to set it higher than 500

[peterjah]

• It would be nice to have the ability to provide a filter when quering keys
• maybe the get_addresses endpoint is not the good place to get storage keys? maybe it would be cleaner to create a new one "get_datastore_keys"
• the "get_datastore_entries" return should also be modified to return a key/value list. because today you provide a list of key in input and receive a list of key in output, but you have to pray that the ordering is kept unchanged and there is no missing values. having key/value pairs as result would be safer and cleaner

[modship]

• QueryState
  • AddressDatastoreKeysCandidate -> Execution.get_final_and_candidate_datastore_keys()
  • AddressDatastoreKeysFinal -> Execution.get_final_and_candidate_datastore_keys()
• massa-api :
  • get_addresses() -> Execution.get_addresses_infos()
• interfaceImpl :
  • get_keys() -> Context.get_keys()
  • get_keys_for() -> Context.get_keys()
  • get_ds_keys_wasmv1() -> Context.get_keys()

---

• Context:
  • get_keys() -> SpeculativeLedger.get_key()

---

• SpeculativeLedger
  • get_keys() -> Ledger.get_datastore_keys()
• Execution
  • get_addresses_infos() -> Execution.get_final_and_candidate_datastore_keys()
  • get_final_and_candidate_datastore_keys() -> Ledger.get_datastore_keys()

[damip]

@modship for now we focus only on the jsonrpc and grpc APIs. Let's not do the ABI part for now

[modship]

@modship for now we focus only on the jsonrpc and grpc APIs. Let's not do the ABI part for now

So, why not just re-using https://github.com/massalabs/massa/blob/main/massa-api-exports/src/page.rs ?
Like

massa/massa-api/src/public.rs

Line 568 in 2a71797

async fn get_stakers(

[damip]

@modship we cannot know the total count nor list all matching keys internally for performance reasons