Make xcm execution non-mandatory

[librelois]

The problem

In the FRAME framework, there are two kind of on-chain execution: mandatory and non-mandatory.

Extrinsic execution is not mandatory, so the block producer can exclude an extrinsic from the block if execution panics. On the other hand, the execution of some hooks and inherents is mandatory, meaning that it is impossible to produce a valid block without executing them.

If a panic occurs during a mandatory execution, the chain is stalled. If this happens to a parachain, it is forced to go through Polkadot governance (or kusama, depending on the network concerned) to deploy a runtime hotfix, which will generally involve several days of downtime for the parachain in question.

For some parachains this is not acceptable, which is why mandatory executions should be minimized.

However, the execution of incoming XCM messages is currently mandatory, which is a problem because XCM messages can execute any runtime call (vwith Transact).
We therefore lose the security feature that discards an extrinsic if it contains a bug.

Today's xcmp queue and dmp queue pallets have been designed in such a way that they cannot easily execute non-mandatory XCM messages.

Deferring execution to the on_idle hook is not a solution, as the on_idle hook is mandatory if there's enough block space left.

Proposal

Create a non-mandatory inherent for pallet xcmp queue and dmp queue, and defer execution of xcm messages in these non-mandatory inherents.

A new Storage item XcmExecuted can store a boolean set to false in on_initialize and verified in on_finalize. If it is still false in on-finalize, then the queue is suspended. Then, to solve the bug, parachain governance must be used.

The goal is precisely to be able to solve the potential buig through the governance of the parachain, without having to involve the governance of the relay chain.

Do you have a better idea of how to achieve this goal?

[Polkadot-Forum]

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/make-xcm-execution-non-mandatory/3294/1

[bkchr]

We therefore lose the security feature that discards an extrinsic if it contains a bug.

Why do you have these extrinsics on your chain in the first place?

And have you seen CallDispatcher that is called for a call to execute. You could reject in there.

[burdges]

We've always wanted some mandatory processing of messages, so parachains cannot screw up whatever messaging demands mandatory execution. I've no idea why XCM should be able to execute any runtime call though, so maybe that should be controlled?

@AlistairStewart has thoughts on non-mandatory delivery or processing of XCMP messages, aka off chain messaging with on-chain signalling. Also, we'd likely encourage some wholly off-chain messaging schemes, with obviously non-mandatory delivery or processing.

[xlc]

Deferring execution to the on_idle hook is not a solution, as the on_idle hook is mandatory if there's enough block space left.

It doesn't need to be. We have implemented mechanism to not executing on_idle hook if something isn't right.

https://github.com/AcalaNetwork/Acala/blob/a35964ee2b375234546108c89c3242c849e0848e/modules/idle-scheduler/src/lib.rs#L114

[burdges]

I'm anyways dubious if the relay chain should be the transport for any non-mandatory messages, aka if XCM should do this. Instead, why not fork XCM into some best effort delivery scheme between parachains which never touches the relay chain?

In other words, can you give a use case where you want the relay chain assurances the message arrived quickly, but accept receiving block producers just choosing to drop a message? If no, then ideally this should happen entirely off the relay chain.

[librelois]

Why do you have these extrinsics on your chain in the first place?

Theoretically, we're supposed to be bug-free, but in practice it's happened in the past, and it's bound to happen again sooner or later. Even if we do our utmost to avoid bugs, it's impossible to guarantee 100% that we'll never have a panic on any extrinsic in the future.

And have you seen CallDispatcher that is called for a call to execute.

We even proposed this trait in the first place, yes it allows filtering, but that's not the subject of the request here.

We've always wanted some mandatory processing of messages, so parachains cannot screw up whatever messaging demands mandatory execution.

This is why the solution I propose is to suspend XCM if execution is not complete. If a collator abuses it to suspend XCM for the wrong reasons, he can be slashed by parachain governance.

I've no idea why XCM should be able to execute any runtime call though, so maybe that should be controlled?

Executing a remote call is one of XCM's most demanded and expected features. If you take that away, XCM loses almost all its appeal.

@AlistairStewart has thoughts on non-mandatory delivery or processing of XCMP messages, aka off chain messaging with on-chain signalling. Also, we'd likely encourage some wholly off-chain messaging schemes, with obviously non-mandatory delivery or processing.

This definitely seems like a good direction to follow for the long term, but it doesn't answer the short-term problem of DMP and HRMP queues.

We have implemented mechanism to not executing on_idle hook if something isn't right.

Thank you very much xlc for sharing, your solution is very interesting, if no collective solution is found in the short term, we will probably implement something similar to you.
But this problem applies to all parachains, so it would be preferable to directly modify the dmp-queue and xmcp-queue pallets so that all parachains can benefit from this security.