Running lightnode as a provider for ApiPromise gets stuck, for a parachain.

[fend25]

Issue Description

Hi!

I'm trying to connect to the Polkadot parachain called Unique, and I'm using the code from the documentation found at:
https://github.com/paritytech/substrate-connect/blob/main/packages/connect/README.md?plain=1#L58-L76

However, the code doesn't reach the initialization finish and gets stuck on the await ApiPromise.create({provider}) expression, with a message printed once per minute, saying:

[smoldot] Smoldot v1.0.11. Current memory usage: 26.3 MiB. Average download: 75.3 kiB/s. Average upload: 646 B/s.

And running almost the same code but for relay chain (I've tested Polkadot and Kusama) it's working properly and not getting stuck.

I'm running this code in Node.js, not in the browser. My goal is to write a block indexer, and my clue is that the light node should speed up scanning significantly.

Here's a gist to reproduce the issue with the code and chain spec: https://gist.github.com/fend25/bcff059ad868f95480c306c7d6ddc87e

Steps to reproduce the issue

1. get chain spec from the gist
2. run the code snippet from the gist

Describe the results you received

> node connect_to_parachain.ts
(node:16044) ExperimentalWarning: Import assertions are not a stable feature of the JavaScript language. Avoid relying on their current behavior and syntax as those might change in a future version of Node.js.
(Use `node --trace-warnings ...` to show where the warning was created)
[smoldot] Smoldot v1.0.11
connect: 273.586ms provider connected
[smoldot] Chain initialization complete for polkadot. Name: "Polkadot". Genesis hash: 0x91b1…90c3. State root hash: 0x29d0d972cd27cbc511e9589fcb7a4506d5eb6a9e8df205f00472e5ab354a4e17. Network identity: 12D3KooWP8beLSQkdmLTyMvRo1CT9D3LDebY5WdRup2g1T7LaKcu. Chain specification or database starting at: 0xdfc5…337d (#16335324)
[json-rpc-polkadot] The JSON-RPC client has just called a JSON-RPC function from the legacy JSON-RPC API (system_health). Legacy JSON-RPC functions have loose semantics and cannot be properly implemented on a light client. You are encouraged to use the new JSON-RPC API <https://github.com/paritytech/json-rpc-interface-spec/> instead. The legacy JSON-RPC API functions will be deprecated and removed in the distant future.
[smoldot] Parachain initialization complete for unq. Name: "UNIQUE". Genesis hash: 0x8432…7d31. State root hash: 0x9fd6243a1015e0550a2a8330315bcced1b0c7625b5fea5a11716e20c8c2e4262. Network identity: 12D3KooWKdjqHKMuu5krjMEj1GB2g6qEhHRYqKAH8vatDbJBnRbL. Relay chain: polkadot (id: 2037)
[json-rpc-unq] The JSON-RPC client has just called a JSON-RPC function from the legacy JSON-RPC API (system_health). Legacy JSON-RPC functions have loose semantics and cannot be properly implemented on a light client. You are encouraged to use the new JSON-RPC API <https://github.com/paritytech/json-rpc-interface-spec/> instead. The legacy JSON-RPC API functions will be deprecated and removed in the distant future.
[sync-service-polkadot] GrandPa warp sync finished to #16443249 (0x46a2…3dcd)
[runtime-polkadot] Finalized block runtime ready. Spec version: 9420. Size of `:code`: 1.4 MiB.
[smoldot] Smoldot v1.0.11. Current memory usage: 26.3 MiB. Average download: 75.3 kiB/s. Average upload: 646 B/s.
[smoldot] Smoldot v1.0.11. Current memory usage: 26.3 MiB. Average download: 18.2 kiB/s. Average upload: 227 B/s.
[smoldot] Smoldot v1.0.11. Current memory usage: 26.4 MiB. Average download: 21.5 kiB/s. Average upload: 257 B/s.
[smoldot] Smoldot v1.0.11. Current memory usage: 26.4 MiB. Average download: 19.0 kiB/s. Average upload: 219 B/s.
[smoldot] Smoldot v1.0.11. Current memory usage: 26.3 MiB. Average download: 19.1 kiB/s. Average upload: 232 B/s.
[smoldot] Smoldot v1.0.11. Current memory usage: 26.4 MiB. Average download: 18.9 kiB/s. Average upload: 248 B/s.
[smoldot] Smoldot v1.0.11. Current memory usage: 26.3 MiB. Average download: 18.7 kiB/s. Average upload: 230 B/s.
[smoldot] Smoldot v1.0.11. Current memory usage: 26.3 MiB. Average download: 20.7 kiB/s. Average upload: 217 B/s.
[smoldot] Smoldot v1.0.11. Current memory usage: 26.3 MiB. Average download: 19.0 kiB/s. Average upload: 257 B/s.

Describe the results you expected

The expected result is that the code finishes the initialization and runs further.

Substrate-Connect version

0.7.30

Provider

Other (Specify below)

Browser version

node.js version 20.3.0

@substrate/[email protected]
@polkadot/[email protected]
@polkadot/[email protected]

Additional environment details

The same result is obtained on Ubuntu 20.04, macOS 13 and Windows 10.

[josepot]

Hi,

Thanks for reporting this issue and for the amount of details provided in it.

I noticed the following in your code-snipped:

const relayProvider = new ScProvider(Sc, Sc.WellKnownChain.polkadot)
const provider = new ScProvider(Sc, parachainSpec, relayProvider)

console.time('connect')

await provider.connect()

could you please try doing this, instead:

const relayProvider = new ScProvider(Sc, Sc.WellKnownChain.polkadot)
const provider = new ScProvider(Sc, parachainSpec, relayProvider)

console.time('connect')

await relayProvider.connect()
await provider.connect()

and let me know if that fixes the issue? 🙏

I know, I know... Why should you first connect to the relay-chain yourself, when the parachain provider knows about the existence of the relay-chain, right? Why doesn't it do it automatically for you?

Great question! I wish I had a good answer for you 😅 In the meanwhile, could you please try that and let me know if that fixes the current issue? 🙏

Thanks!

EDIT:

FWIW: I'm don't think that what I've suggested will fix this issue. So, I'm going to first try and see if this is an issue with the ScProvider (which is part of polkadotjs) or with substrate-connect.

[josepot]

Hi @fend25 !

So, I've did some digging and there seems to be an issue with the bootnodes of your chainspec? b/c smoldot is unable to find any peers. Later today, I will try to recreate the issue using smoldot directly, just to make sure that this is not an issue with substrate-connect.

Please have a look at this stackblitz demo, which produces the following output:

[fend25]

Hi @josepot! Thank you for such warm response!

Adding await relayProvider.connect() did not help; it continues to print "[smoldot]..." messages and does not subscribe to new heads.

[josepot]

Hi @josepot! Thank you for such warm response!

Adding await relayProvider.connect() did not help; it continues to print "[smoldot]..." messages and does not subscribe to new heads.

Hi @fend25 ! Did you have a chance to review see my second response? 🙏

[fend25]

Yes, the second option works, but it only provides an RPC connection, as far as I can see. The initial idea was to seamlessly use the Polkadot.js library. However, given that I still require a fully synchronized node on the same host to quickly access historic data, making an HTTP request to localhost might not add significantly more overhead compared to using a light node. But it will require to manually parse encoded data from storage (for extrinsic args for example)...