Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

evm data sourcing #911

Merged
merged 13 commits into from
Feb 20, 2024
30 changes: 18 additions & 12 deletions docs/concepts/trading-on-vega/data-sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@ title: Data sourcing
hide_title: false
---

import NetworkParameter from '@site/src/components/NetworkParameter';

Vega's data sourcing framework enables the Vega protocol to acquire and consume data, for example, to settle a market, or to terminate trading at a market's expiry.

It accepts data from several source types, including Ethereum, off-chain data signed by a known key, and data from Vega itself.
It accepts data from several source types, including Ethereum mainnet, EVM chains that support Ethereum RPC calls, off-chain data signed by a known key, and data from Vega itself.

The data sourcing framework can also process data from specific, selected fields out of a larger data object, filtering out irrelevant or potentially erroneous data.

Expand All @@ -23,6 +25,7 @@ For a market proposer looking to choose which data source is best for their mark

## Sources of data
Inputs to the data sourcing framework can come from:
* Any chain or layer 2 chain that supports Ethereum RPC calls and runs an EVM
* Ethereum smart contracts
* Signed message data source, part of the [Open Oracle ↗](https://github.com/compound-finance/open-oracle) feed
* Specially formatted and signed JSON messages
Expand All @@ -46,27 +49,30 @@ Data sources must be able to emit the following data types:
* Date/Time - to compare against in filters
* Structured data records - such as a set of key and value pairs (inputs to filters)

## Ethereum data sources
Ethereum oracles bridge Ethereum based data sources such as [Chainlink ↗](https://chain.link/) and [UMA ↗](https://uma.xyz/) in to Vega, enabling markets to be settled or priced using data that is verified on Ethereum. The interface is flexible enough that it allows for any Ethereum smart contract to be used as a data source.
## EVM data sources
EVM oracles bridge Ethereum- or any EVM-based data sources in to Vega, enabling markets to be settled or priced using data that is verified on a chain that supports Ethereum RPC calls.

Each chain needs to be supported in the network parameter:
<NetworkParameter frontMatter={frontMatter} param="blockchains.ethereumRpcAndEvmCompatDataSourcesConfig" hideValue="true" />, and by the validators running the network. Use a [network parameter proposal](../../tutorials/proposals/network-parameter-proposal.md) to get a new data source chain added to the network.
candida-d marked this conversation as resolved.
Show resolved Hide resolved

:::note
Currently data can only be read from a smart contract based on a timed trigger. In future Vega will also support events emitted by contracts.
Currently data can only be read from a smart contract based on a timed trigger.
candida-d marked this conversation as resolved.
Show resolved Hide resolved
candida-d marked this conversation as resolved.
Show resolved Hide resolved
:::

When the contract call is triggered by the data source, Vega validator nodes read the selected data from Ethereum and submit a transaction that includes the filtered data. When that data is verified by enough validators, the market's data source specification then acts on the submitted data.
When the contract call is triggered by the data source, Vega validator nodes read the selected data from the chain and submit a transaction that includes the filtered data. When that data is verified by enough validators, the market's data source specification then acts on the submitted data.

An Ethereum data source specification must include:
An EVM data source specification must include:

- Ethereum contract address
- Contract address
- ABI in JSON format for the contract (or a subset, covering the parts relevant to fetching data)
- Name of the function call, along with any parameters that must be passed through

This data can be used for filters, or used as the oracle data itself.

All data sourced from Ethereum is structured as an object containing both a payload and Ethereum chain metadata. Specifically:
All data sourced from Ethereum and EVM chains is structured as an object containing both a payload and chain metadata. Specifically:

- Ethereum block height at which the data was observed/event occurred
- Ethereum block timestamp when the data was observed/event occurred
- Block height at which the data was observed/event occurred
- Block timestamp when the data was observed/event occurred

## Signed message data sources
Signed message data sources are a source of off-chain data. They introduce a Vega transaction that represents a data result that is validated by ensuring the signed message is provided by the Vega or Ethereum public key provided in the market’s proposal.
Expand Down Expand Up @@ -160,12 +166,12 @@ Data source filters allow the market proposer to specify, for a given "root" dat

Products on Vega use data to drive actions like settlement and to progress through the market lifecycle. The two key transitions controlled by data sources for cash settled futures are the termination of trading, and settlement of the market. For cash settled perpetuals markets, the data source provides prices for the settlement schedule. These are configured when the market is proposed by providing a data source specification that covers the root source and any filters required to select the specific data or trigger event.

As an Ethereum contract or a public key may provide many messages, a filter is used to extract the required message - for example trading on a futures market could terminate at a specific date and time, and so the filters would ensure that only data provided on or after the specified date and time would trigger termination.
As a smart contract or a public key may provide many messages, a filter is used to extract the required message - for example trading on a futures market could terminate at a specific date and time, and so the filters would ensure that only data provided on or after the specified date and time would trigger termination.

When a market is proposed, the market proposal must include details for filters to be applied to the chosen data source(s). Those filters are applied to the source of structured data records that are used as input and determine how data is emitted: such as the specific value for a named field, to return `BTCUSD_PRICE` from a record containing many prices, for example; or price data on/after a certain time.

## Submitting data for a market
When using an Ethereum oracle, there is no need to submit data once the Ethereum oracle is specified in the market proposal.
When using an EVM oracle, there is no need to submit data once the EVM oracle is specified in the market proposal.

For other data source types, any Vega keypair can submit settlement and market termination data to the chain. The creator of an instrument for a market has chosen in advance a price source, which data fields the market requires to settle and terminate, and filters that determine when the data is used.

Expand Down
1 change: 1 addition & 0 deletions docs/tutorials/proposals/new-market-proposal.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@ An instrument contains the following properties:
| `dataSourceSpecForSettlementData` | This defines the data source that will be used to identify the settlement price when the market expires. | prices.ORANGES.value |
| `dataSourceSpecForTradingTermination` | The fields that define the data source used for terminating trading on the market. | vegaprotocol.builtin.timestamp |
| `dataSourceSpecBinding` | The fields describe how specific information provided by the data source is used. For example, they can identify the specific name of the settlement price output, or the specific name of the trading termination property. |
| `sourceChainId` | Describes the chain ID of the data source. This chain must already be enabled in network parameters and supported by validators. |
candida-d marked this conversation as resolved.
Show resolved Hide resolved


For easy reading, the data source filters are separated out - see [Data source bindings](#data-source-bindings) below to see the fields for specifying data.
Expand Down
1 change: 1 addition & 0 deletions docs/tutorials/proposals/new-perpetuals-market.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ An instrument contains the following properties:
| [`dataSourceSpecForSettlementData`](#data-source-for-settlement-data) | This defines the Ethereum data source, the method, normalisers, required confirmations, etc, that will be used to identify the settlement price when the market expires. | |
| [`dataSourceSpecForSettlementSchedule`](#data-source-for-settlement-schedule) | This defines how the market will source data for funding, and how often to source it. | |
| [`dataSourceSpecBinding`](#data-source-bindings) | The fields describe how specific information provided by the data source is used. For example, they are used to set the settlement data property and the settlement schedule property. |
| `sourceChainId` | Describes the chain ID of the data source. This chain must already be enabled in network parameters and supported by validators. |

### Data source for settlement schedule
The periodic settlements scheduled with the fields below determine how often the market's funding payments occur. It's recommended that funding payments are be less frequent than auction extensions for [price monitoring](#price-monitoring). Very frequent funding payments may lead to quick price changes in the market that participants may not have time to react to. Setting longer funding payment triggers allow for more time.
Expand Down
27 changes: 16 additions & 11 deletions docs/tutorials/using-data-sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,13 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

## Choosing a data source when proposing a market
A market proposal must specify details about the data it requires in the market creation governance proposal. When configuring a market's instrument, you will need to select the data sourcefor two events: settlement and trading termination.
A market proposal must specify details about the data it requires. When configuring a market's instrument, you will need to select the data source for the following events: settlement and, if applicable, trading termination.

This is done by:
1. Defining a data source spec binding for settlement price
2. Configuring a data source spec for settlement price values
3. Defining a data source spec binding for trading termination
4. Configuring a data source spec for trading termination values
3. Defining a data source spec binding for trading termination, for dated futures markets
4. Configuring a data source spec for trading termination values, for dated futures markets

The **binding** tells the market which field contains the value. The **spec** defines where this data will come from, and which values to pass through to the binding.

Expand All @@ -24,13 +24,16 @@ The **binding** tells the market which field contains the value. The **spec** de
* [Tutorial: Proposing a market](./proposals/new-market-proposal.md)
:::

## Ethereum oracles
Since version 0.73.0, settlement data can be sourced from Ethereum smart contracts. The following spec would read from the Ethereum contract at `0x1b4...e43` every 30 seconds, and fetch the Bitcoin price value from the returned object:
## EVM data sources
Settlement data can be sourced from smart contracts and EVM chains that support Ethereum RPC calls.

The following spec would read from the Ethereum contract at `0x1b4...e43` every 30 seconds, and fetch the Bitcoin price value from the returned object:

```javascript
"dataSourceSpecForSettlementData": {
"external": {
"ethOracle": {
"sourceChainId": "1",
"address": "0x1b44F3514812d835EB1BDB0acB33d3fA3351Ee43",
"abi": "[{\"inputs\":[],\"name\":\"latestAnswer\",\"outputs\":[{\"internalType\":\"int256\",\"name\":\"\",\"type\":\"int256\"}],\"stateMutability\":\"view\",\"type\":\"function\"}]",
"method": "latestAnswer",
Expand Down Expand Up @@ -63,10 +66,12 @@ Since version 0.73.0, settlement data can be sourced from Ethereum smart contrac
}
```

This will cause the network validators to read the result from the specified smart contract and submit the result to Vega. When the data is verified by enough validators, this price is accepted on to the network. This is unlike the other two Oracle types ([Open Oracle signed messages](#open-oracle-signed-messages) & [JSON signed message data](#json-signed-message-data)),which both rely on a third party submitting data to the network.
This will cause the network validators to read the result from the specified smart contract and submit the result to Vega. When the data is verified by enough validators, this price is accepted on to the network.

This is unlike the other two Oracle types ([Open Oracle signed messages](#open-oracle-signed-messages) & [JSON signed message data](#json-signed-message-data)),which both rely on a third party submitting data to the network.

### ABI
The `address` field in the specification tells the spec above which address to interact with on Ethereum. The `abi` ([Application Binary Interface](https://docs.soliditylang.org/en/develop/abi-spec.html#json) & `method` field on the spec above tells the settlement spec **how** to interact with it. Ethereum Oracle settlement specifications use the JSON ABI of the smart contract to describe the method on the contract that will be called to fetch the data. The ABI will contain the function name, details of any paramters required, and the format of the response.
The `address` field in the specification tells the spec above which address to interact with on the chain. The `abi` ([Application Binary Interface](https://docs.soliditylang.org/en/develop/abi-spec.html#json) & `method` field on the spec above tells the settlement spec **how** to interact with it. EVM oracle settlement specifications use the JSON ABI of the smart contract to describe the method on the contract that will be called to fetch the data. The ABI will contain the function name, details of any paramters required, and the format of the response.

For example, the [Chainlink BTC/USD oracle](https://data.chain.link/ethereum/mainnet/crypto-usd/btc-usd) has its JSON ABI [published on Etherscan](https://etherscan.io/address/0xf4030086522a5beea4988f8ca5b36dbc97bee88c#code). When defining the data source spec, you can populate the `abi` field with the full ABI, and then set the `method` to `latestAnswer`.

Expand All @@ -75,15 +80,15 @@ When populating the `abi` field on your data source spec, you can remove the met
:::

### Time trigger
As it says above, with Ethereum data source specs the validators will read the specified smart contract and method detailed in the ABI. The `trigger` instructs the validators when to do this. In the smple above, it will be called every 30 seconds.
As it says above, with EVM data source specs the validators will read the specified smart contract and method detailed in the ABI. The `trigger` instructs the validators when to do this. In the smple above, it will be called every 30 seconds.

### Normaliser
A [JSONPath](https://datatracker.ietf.org/wg/jsonpath/about/) expression use to extract data from the JSON returned from the method call. In the spec above, `expression` is set to `$[0]`, which returns the first item in an array. `$` would return the complete result.

## Open Oracle signed messages
Vega's Data Sourcing framework supports signed ABI-encoded [Open Oracle ↗](https://github.com/compound-finance/open-oracle) or JSON messages. ABI-encoded signed messages can be verified to have come from the public key that signed them, which allows markets on Vega to use pricing data sourced from Ethereum.
Vega's data sourcing framework supports signed ABI-encoded [Open Oracle ↗](https://github.com/compound-finance/open-oracle) or JSON messages. ABI-encoded signed messages can be verified to have come from the public key that signed them, which allows markets on Vega to use pricing data sourced from Ethereum.

When it's time for a market to settle, someone needs to submit the data that matches the data source spec defined in the market. Any Vega keypair can submit the data. In the configuration for a market, a data source specification field dictates which data feeds it is interested in. In effect, it works as a filter. This specification means that the creator of an instrument for a market will choose in advance a price source, and which data fields the market requires to settle and terminate.
When it's time for a market to settle, someone needs to submit the data that matches the data source spec defined in the market. Any Vega keypair can submit the data. In the configuration for a market, a data source specification field dictates which data feeds it is interested in. In effect, it works as a filter. This specification means that the creator of an instrument for a market will choose in advance a price source, and which data fields the market requires to settle and optionally terminate.

### Using Open Oracle signed messages in a market proposal
For the binding, use the `name` field of the data. In the case of Open Oracle messages, the price data will be availableas 'prices.currency-code.value', for example:`"prices.BTC.value"`.
Expand Down Expand Up @@ -208,7 +213,7 @@ The [Oracle Data list REST endpoint](../api/rest/data-v2/trading-data-service-li
## JSON signed message data
[JSON ↗](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON) messages are a simpler, more configurable alternative to Open Oracle data. They can be totally custom objects, as long as they are valid JSON. As they are not attested by any off-chain source in the way that Open Oracle messages are, and so it's generally advisable to check for an Open Oracle price source before choosing JSON data. The Vega key that signs the message will be referred to as the source for the data.

When it's time for a market to settle, someone needs to submit the data that matches the data source spec defined in the market. Any Vega keypair can submit the data. In the configuration for a market, a data source specification field dictates which data feeds it is interested in. In effect, it works as a filter. This specification means that the creator of an instrument for a market will choose in advance a price source, and which data fields the market requires to settle and terminate.
When it's time for a market to settle, someone needs to submit the data that matches the data source spec defined in the market. Any Vega keypair can submit the data. In the configuration for a market, a data source specification field dictates which data feeds it is interested in. In effect, it works as a filter. This specification means that the creator of an instrument for a market will choose in advance a price source, and which data fields the market requires to settle and optionally terminate.

### Using JSON signed message data in a market proposal
For the binding, use the `name` field of the data. In the following example, the market is settled based on the number of people who have walked on the moon.
Expand Down