Skip to content

Commit 9b6ae38

Browse files
evan-forbesevan-forbes
committedJul 26, 2022
add remaining ADRs
·
v2.0.0-tm-v0.37.2-rc01.5.0-tm-v0.34.20-cat-4
1 parent 3354f6b commit 9b6ae38

10 files changed

+1158
-0
lines changed
 

‎docs/celestia-architecture/adr-001-block-propagation.md

+124
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
# ADR 001: Erasure Coding Block Propagation
2+
3+
## Changelog
4+
5+
- 16-2-2021: Created
6+
7+
## Context
8+
9+
Block propagation is currently done by splitting the block into arbitrary chunks and gossiping them to validators via a gossip routine. While this does not have downsides it does not meet the needs of the Celestia chain. The celestia chain requires blocks to be encoded in a different way and for the proposer to not propagate the chunks to peers.
10+
11+
Celestia wants validators to pull the block from a IPFS network. What does this mean? As I touched on earlier the proposer pushes the block to the network, this in turn means that each validator downloads and reconstructs the block each time to verify it. Instead Celestia will encode and split up the block via erasure codes, stored locally in the nodes IPFS daemon. After the proposer has sent the block to IPFS and received the CIDs it will include them into the proposal. This proposal will be gossiped to other validators, once a validator receives the proposal it will begin requesting the CIDs included in the proposal.
12+
13+
There are two forms of a validator, one that downloads the block and one that samples it. What does sampling mean? Sampling is the act of checking that a portion or entire block is available for download.
14+
15+
## Detailed Design
16+
17+
The proposed design is as follows.
18+
19+
### Types
20+
21+
The proposal and vote types have a BlockID, this will be replaced with a header hash. The proposal will contain add fields.
22+
23+
The current proposal will be updated to include required fields. The entirety of the message will be reworked at a later date. To see the extent of the needed changes you can visit the [spec repo](https://github.com/celestiaorg/celestia-specs/blob/master/specs/proto/consensus.proto#L19)
24+
25+
```proto
26+
message Proposal {
27+
SignedMsgType type = 1;
28+
int64 height = 2;
29+
int32 round = 3;
30+
int32 pol_round = 4;
31+
32+
+++
33+
// 32-byte hash
34+
bytes last_header_hash = 5;
35+
// 32-byte hash
36+
bytes last_commit_hash = 6;
37+
// 32-byte hash
38+
bytes consensus_root = 7;
39+
FeeHeader fee_header = 8;
40+
// 32-byte hash
41+
bytes state_commitment = 9;
42+
uint64 available_data_original_shares_used = 10;
43+
AvailableDataHeader available_data_header = 11;
44+
+++
45+
46+
google.protobuf.Timestamp timestamp = 12
47+
[(gogoproto.nullable) = false, (gogoproto.stdtime) = true];
48+
bytes signature = 12;
49+
}
50+
```
51+
52+
```proto
53+
// Vote represents a prevote, precommit, or commit vote from validators for
54+
// consensus.
55+
message Vote {
56+
SignedMsgType type = 1;
57+
int64 height = 2;
58+
int32 round = 3;
59+
+++
60+
bytes header_hash = 4;
61+
+++
62+
google.protobuf.Timestamp timestamp = 5
63+
[(gogoproto.nullable) = false, (gogoproto.stdtime) = true];
64+
bytes validator_address = 6;
65+
int32 validator_index = 7;
66+
bytes signature = 8;
67+
}
68+
```
69+
70+
See [specs](https://github.com/celestiaorg/celestia-specs/blob/master/specs/data_structures.md#vote) for more details on the vote.
71+
72+
### Disk Storage
73+
74+
Currently celestia-core stores all blocks in its store. Going forward only the headers of the blocks within the unbonding period will be stored. This will drastically reduce the amount of storage required by a celestia-core node. After the unbonding period all headers will have the option of being pruned.
75+
76+
Proposed amendment to `BlockStore` interface
77+
78+
```go
79+
type BlockStore interface {
80+
Base() int64
81+
Height() int64
82+
Size() int64
83+
84+
LoadBlockMeta(height int64) *types.BlockMeta
85+
LoadHeader(height int64) *types.Header
86+
LoadDAHeader(height int64) *types.DataAvailabilityHeader
87+
88+
SaveHeaders(header *types.Header, daHeader *types.DataAvailabilityHeader, seenCommit *types.Commit)
89+
90+
PruneHeaders(height int64) (uint64, error)
91+
92+
LoadBlockCommit(height int64) *types.Commit
93+
LoadSeenCommit(height int64) *types.Commit
94+
}
95+
```
96+
97+
Along side these changes the rpc layer will need to change. Instead of querying the LL-core store, the node will redirect the query through IPFS.
98+
99+
Example:
100+
101+
When a user requests a block from the LL node, the request will be set to the IPLD plugin. If the IPLD does not have the requested block, it will make a request to the celestia IPFS network for the required CIDs. If the full node does not have the DAheader they will not be able to request the block data.
102+
103+
![user request flow](./assets/user-request.png)
104+
105+
The goal is to not change the public interface for RPC's. It is yet to be seen if this possible. This means that CIDs will need to be set and loaded from the store in order to get all the related block information an user requires.
106+
107+
## Status
108+
109+
Proposed
110+
111+
112+
### Positive
113+
114+
- Minimal breakage to public interface
115+
- Only store the block in a single place (IPFS)
116+
- Reduce the public interface of the storage within Celestia.
117+
118+
### Negative
119+
120+
- User requests may take more time to process
121+
122+
### Neutral
123+
124+
## References

‎docs/celestia-architecture/adr-002-ipld-da-sampling.md

+280
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,280 @@
1+
# ADR 002: Sampling erasure coded Block chunks
2+
3+
## Changelog
4+
5+
- 26-2-2021: Created
6+
7+
## Context
8+
9+
In Tendermint's block gossiping each peer gossips random parts of block data to peers.
10+
For Celestia, we need nodes (from light-clients to validators) to be able to sample row-/column-chunks of the erasure coded
11+
block (aka the extended data square) from the network.
12+
This is necessary for Data Availability proofs.
13+
14+
![extended_square.png](img/extended_square.png)
15+
16+
A high-level, implementation-independent formalization of above mentioned sampling and Data Availability proofs can be found in:
17+
[_Fraud and Data Availability Proofs: Detecting Invalid Blocks in Light Clients_](https://fc21.ifca.ai/papers/83.pdf).
18+
19+
For the time being, besides the academic paper, no other formalization or specification of the protocol exists.
20+
Currently, the Celestia specification itself only describes the [erasure coding](https://github.com/celestiaorg/celestia-specs/blob/master/specs/data_structures.md#erasure-coding)
21+
and how to construct the extended data square from the block data.
22+
23+
This ADR:
24+
- describes the high-level requirements
25+
- defines the API that and how it can be used by different components of Celestia (block gossiping, block sync, DA proofs)
26+
- documents decision on how to implement this.
27+
28+
29+
The core data structures and the erasure coding of the block are already implemented in celestia-core ([#17], [#19], [#83]).
30+
While there are no ADRs for these changes, we can refer to the Celestia specification in this case.
31+
For this aspect, the existing implementation and specification should already be on par for the most part.
32+
The exact arrangement of the data as described in this [rationale document](https://github.com/celestiaorg/celestia-specs/blob/master/rationale/message_block_layout.md)
33+
in the specification can happen at app-side of the ABCI boundary.
34+
The latter was implemented in [celestiaorg/celestia-app#21](https://github.com/celestiaorg/celestia-app/pull/21)
35+
leveraging a new ABCI method, added in [#110](https://github.com/celestiaorg/celestia-core/pull/110).
36+
This new method is a sub-set of the proposed ABCI changes aka [ABCI++](https://github.com/tendermint/spec/pull/254).
37+
38+
Mustafa Al-Bassam (@musalbas) implemented a [prototype](https://github.com/celestiaorg/celestia-prototype)
39+
whose main purpose is to realistically analyse the protocol.
40+
Although the prototype does not make any network requests and only operates locally, it can partly serve as a reference implementation.
41+
It uses the [rsmt2d] library.
42+
43+
The implementation will essentially use IPFS' APIs. For reading (and writing) chunks it
44+
will use the IPLD [`DagService`](https://github.com/ipfs/go-ipld-format/blob/d2e09424ddee0d7e696d01143318d32d0fb1ae63/merkledag.go#L54),
45+
more precisely the [`NodeGetter`](https://github.com/ipfs/go-ipld-format/blob/d2e09424ddee0d7e696d01143318d32d0fb1ae63/merkledag.go#L18-L27)
46+
and [`NodeAdder`](https://github.com/ipfs/go-ipld-format/blob/d2e09424ddee0d7e696d01143318d32d0fb1ae63/merkledag.go#L29-L39).
47+
As an optimization, we can also use a [`Batch`](https://github.com/ipfs/go-ipld-format/blob/d2e09424ddee0d7e696d01143318d32d0fb1ae63/batch.go#L29)
48+
to batch adding and removing nodes.
49+
This will be achieved by passing around a [CoreAPI](https://github.com/ipfs/interface-go-ipfs-core/blob/b935dfe5375eac7ea3c65b14b3f9a0242861d0b3/coreapi.go#L15)
50+
object, which derive from the IPFS node which is created along a with a tendermint node (see [#152]).
51+
This code snippet does exactly that (see the [go-ipfs documentation] for more examples):
52+
```go
53+
// This constructs an IPFS node instance
54+
node, _ := core.NewNode(ctx, nodeOptions)
55+
// This attaches the Core API to the constructed node
56+
coreApi := coreapi.NewCoreAPI(node)
57+
```
58+
59+
The above mentioned IPLD methods operate on so called [ipld.Nodes].
60+
When computing the data root, we can pass in a [`NodeVisitor`](https://github.com/celestia/nmt/blob/b22170d6f23796a186c07e87e4ef9856282ffd1a/nmt.go#L22)
61+
into the Namespaced Merkle Tree library to create these (each inner- and leaf-node in the tree becomes an ipld node).
62+
As a peer that requests such an IPLD node, the Celestia IPLD plugin provides the [function](https://github.com/celestiaorg/celestia-core/blob/ceb881a177b6a4a7e456c7c4ab1dd0eb2b263066/p2p/ipld/plugin/nodes/nodes.go#L175)
63+
`NmtNodeParser` to transform the retrieved raw data back into an `ipld.Node`.
64+
65+
A more high-level description on the changes required to rip out the current block gossiping routine,
66+
including changes to block storage-, RPC-layer, and potential changes to reactors is either handled in [ADR 001](./adr-001-block-propagation.md),
67+
and/or in a few smaller, separate followup ADRs.
68+
69+
## Alternative Approaches
70+
71+
Instead of creating a full IPFS node object and passing it around as explained above
72+
- use API (http)
73+
- use ipld-light
74+
- use alternative client
75+
76+
Also, for better performance
77+
- use [graph-sync], [IPLD selectors], e.g. via [ipld-prime]
78+
79+
Also, there is the idea, that nodes only receive the [Header] with the data root only
80+
and, in an additional step/request, download the DA header using the library, too.
81+
While this feature is not considered here, and we assume each node that uses this library has the DA header, this assumption
82+
is likely to change when flesh out other parts of the system in more detail.
83+
Note that this also means that light clients would still need to validate that the data root and merkelizing the DA header yield the same result.
84+
85+
## Decision
86+
87+
> This section records the decision that was made.
88+
> It is best to record as much info as possible from the discussion that happened. This aids in not having to go back to the Pull Request to get the needed information.
89+
90+
> - TODO: briefly summarize github, discord, and slack discussions (?)
91+
> - also mention Mustafa's prototype and compare both apis briefly (RequestSamples, RespondSamples, ProcessSamplesResponse)
92+
> - mention [ipld experiments]
93+
94+
95+
96+
## Detailed Design
97+
98+
Add a package to the library that provides the following features:
99+
1. sample a given number of random row/col indices of extended data square given a DA header and indicate if successful or timeout/other error occurred
100+
2. store the block in the network by adding it to the peer's local Merkle-DAG whose content is discoverable via a DHT
101+
3. store the sampled chunks in the network
102+
4. reconstruct the whole block from a given DA header
103+
5. get all messages of a particular namespace ID.
104+
105+
We mention 5. here mostly for completeness. Its details will be described / implemented in a separate ADR / PR.
106+
107+
Apart from the above mentioned features, we informally collect additional requirements:
108+
- where randomness is needed, the randomness source should be configurable
109+
- all replies by the network should be verified if this is not sufficiently covered by the used libraries already (IPFS)
110+
- where possible, the requests to the network should happen in parallel (without DoSing the proposer for instance).
111+
112+
This library should be implemented as two new packages:
113+
114+
First, a sub-package should be added to the layzledger-core [p2p] package
115+
which does not know anything about the core data structures (Block, DA header etc).
116+
It handles the actual network requests to the IPFS network and operates on IPFS/IPLD objects
117+
directly and hence should live under [p2p/ipld].
118+
To a some extent this part of the stack already exists.
119+
120+
Second, a high-level API that can "live" closer to the actual types, e.g., in a sub-package in [celestia-core/types]
121+
or in a new sub-package `da`.
122+
123+
We first describe the high-level library here and describe functions in
124+
more detail inline with their godoc comments below.
125+
126+
### API that operates on celestia-core types
127+
128+
As mentioned above this part of the library has knowledge of the core types (and hence depends on them).
129+
It does not deal with IPFS internals.
130+
131+
```go
132+
// ValidateAvailability implements the protocol described in https://fc21.ifca.ai/papers/83.pdf.
133+
// Specifically all steps of the protocol described in section
134+
// _5.2 Random Sampling and Network Block Recovery_ are carried out.
135+
//
136+
// In more detail it will first create numSamples random unique coordinates.
137+
// Then, it will ask the network for the leaf data corresponding to these coordinates.
138+
// Additionally to the number of requests, the caller can pass in a callback,
139+
// which will be called on for each retrieved leaf with a verified Merkle proof.
140+
//
141+
// Among other use-cases, the callback can be useful to monitoring (progress), or,
142+
// to process the leaf data the moment it was validated.
143+
// The context can be used to provide a timeout.
144+
// TODO: Should there be a constant = lower bound for #samples
145+
func ValidateAvailability(
146+
ctx contex.Context,
147+
dah *DataAvailabilityHeader,
148+
numSamples int,
149+
onLeafValidity func(namespace.PrefixedData8),
150+
) error { /* ... */}
151+
152+
// RetrieveBlockData can be used to recover the block Data.
153+
// It will carry out a similar protocol as described for ValidateAvailability.
154+
// The key difference is that it will sample enough chunks until it can recover the
155+
// full extended data square, including original data (e.g. by using rsmt2d.RepairExtendedDataSquare).
156+
func RetrieveBlockData(
157+
ctx contex.Context,
158+
dah *DataAvailabilityHeader,
159+
api coreiface.CoreAPI,
160+
codec rsmt2d.Codec,
161+
) (types.Data, error) {/* ... */}
162+
163+
// PutBlock operates directly on the Block.
164+
// It first computes the erasure coding, aka the extended data square.
165+
// Row by row ir calls a lower level library which handles adding the
166+
// the row to the Merkle Dag, in our case a Namespaced Merkle Tree.
167+
// Note, that this method could also fill the DA header.
168+
// The data will be pinned by default.
169+
func (b *Block) PutBlock(ctx contex.Context, nodeAdder ipld.NodeAdder) error
170+
```
171+
172+
We now describe the lower-level library that will be used by above methods.
173+
Again we provide more details inline in the godoc comments directly.
174+
175+
`PutBlock` is a method on `Block` as the erasure coding can then be cached, e.g. in a private field
176+
in the block.
177+
178+
### Changes to the lower level API closer to IPFS (p2p/ipld)
179+
180+
```go
181+
// GetLeafData takes in a Namespaced Merkle tree root transformed into a Cid
182+
// and the leaf index to retrieve.
183+
// Callers also need to pass in the total number of leaves of that tree.
184+
// Internally, this will be translated to a IPLD path and corresponds to
185+
// an ipfs dag get request, e.g. namespacedCID/0/1/0/0/1.
186+
// The retrieved data should be pinned by default.
187+
func GetLeafData(
188+
ctx context.Context,
189+
rootCid cid.Cid,
190+
leafIndex uint32,
191+
totalLeafs uint32, // this corresponds to the extended square width
192+
api coreiface.CoreAPI,
193+
) ([]byte, error)
194+
```
195+
196+
`GetLeafData` can be used by above `ValidateAvailability` and `RetrieveBlock` and
197+
`PutLeaves` by `PutBlock`.
198+
199+
### A Note on IPFS/IPLD
200+
201+
In IPFS all data is _content addressed_ which basically means the data is identified by its hash.
202+
Particularly, in the Celestia case, the root CID identifies the Namespaced Merkle tree including all its contents (inner and leaf nodes).
203+
This means that if a `GetLeafData` request succeeds, the retrieved leaf data is in fact the leaf data in the tree.
204+
We do not need to additionally verify Merkle proofs per leaf as this will essentially be done via IPFS on each layer while
205+
resolving and getting to the leaf data.
206+
207+
> TODO: validate this assumption and link to code that shows how this is done internally
208+
209+
### Implementation plan
210+
211+
As fully integrating Data Available proofs into tendermint, is a rather larger change we break up the work into the
212+
following packages (not mentioning the implementation work that was already done):
213+
214+
1. Flesh out the changes in the consensus messages ([celestia-specs#126], [celestia-specs#127])
215+
2. Flesh out the changes that would be necessary to replace the current block gossiping ([ADR 001](./adr-001-block-propagation.md))
216+
3. Add the possibility of storing and retrieving block data (samples or whole block) to celestia-core (this ADR and related PRs).
217+
4. Integrate above API (3.) as an addition into celestia-core without directly replacing the tendermint counterparts (block gossip etc).
218+
5. Rip out each component that will be redundant with above integration in one or even several smaller PRs:
219+
- block gossiping (see ADR 001)
220+
- modify block store (see ADR 001)
221+
- make downloading full Blocks optional (flag/config)
222+
- route some RPC requests to IPFS (see ADR 001)
223+
224+
225+
## Status
226+
227+
Proposed
228+
229+
## Consequences
230+
231+
### Positive
232+
233+
- simplicity & ease of implementation
234+
- can re-use an existing networking and p2p stack (go-ipfs)
235+
- potential support of large, cool, and helpful community
236+
- high-level API definitions independent of the used stack
237+
238+
### Negative
239+
240+
- latency
241+
- being connected to the public IPFS network might be overkill if peers should in fact only care about a subset that participates in the Celestia protocol
242+
- dependency on a large code-base with lots of features and options of which we only need a small subset of
243+
244+
### Neutral
245+
- two different p2p layers exist in celestia-core
246+
247+
## References
248+
249+
- https://github.com/celestiaorg/celestia-core/issues/85
250+
- https://github.com/celestiaorg/celestia-core/issues/167
251+
252+
- https://docs.ipld.io/#nodes
253+
- https://arxiv.org/abs/1809.09044
254+
- https://fc21.ifca.ai/papers/83.pdf
255+
- https://github.com/tendermint/spec/pull/254
256+
257+
258+
[#17]: https://github.com/celestiaorg/celestia-core/pull/17
259+
[#19]: https://github.com/celestiaorg/celestia-core/pull/19
260+
[#83]: https://github.com/celestiaorg/celestia-core/pull/83
261+
262+
[#152]: https://github.com/celestiaorg/celestia-core/pull/152
263+
264+
[celestia-specs#126]: https://github.com/celestiaorg/celestia-specs/issues/126
265+
[celestia-specs#127]: https://github.com/celestiaorg/celestia-specs/pulls/127
266+
[Header]: https://github.com/celestiaorg/celestia-specs/blob/master/specs/data_structures.md#header
267+
268+
[go-ipfs documentation]: https://github.com/ipfs/go-ipfs/tree/master/docs/examples/go-ipfs-as-a-library#use-go-ipfs-as-a-library-to-spawn-a-node-and-add-a-file
269+
[ipld experiments]: https://github.com/celestia/ipld-plugin-experiments
270+
[ipld.Nodes]: https://github.com/ipfs/go-ipld-format/blob/d2e09424ddee0d7e696d01143318d32d0fb1ae63/format.go#L22-L45
271+
[graph-sync]: https://github.com/ipld/specs/blob/master/block-layer/graphsync/graphsync.md
272+
[IPLD selectors]: https://github.com/ipld/specs/blob/master/selectors/selectors.md
273+
[ipld-prime]: https://github.com/ipld/go-ipld-prime
274+
275+
[rsmt2d]: https://github.com/celestia/rsmt2d
276+
277+
278+
[p2p]: https://github.com/celestiaorg/celestia-core/tree/0eccfb24e2aa1bb9c4428e20dd7828c93f300e60/p2p
279+
[p2p/ipld]: https://github.com/celestiaorg/celestia-core/tree/0eccfb24e2aa1bb9c4428e20dd7828c93f300e60/p2p/ipld
280+
[celestia-core/types]: https://github.com/celestiaorg/celestia-core/tree/0eccfb24e2aa1bb9c4428e20dd7828c93f300e60/types

‎docs/celestia-architecture/adr-003-application-data-retrieval.md

+141
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
# ADR 003: Retrieving Application messages
2+
3+
## Changelog
4+
5+
- 2021-04-25: initial draft
6+
7+
## Context
8+
9+
This ADR builds on top of [ADR 002](adr-002-ipld-da-sampling.md) and will use the implemented APIs described there.
10+
The reader should familiarize themselves at least with the high-level concepts the as well as in the [specs](https://github.com/celestiaorg/celestia-specs/blob/master/specs/data_structures.md#2d-reed-solomon-encoding-scheme).
11+
12+
The academic [paper](https://arxiv.org/abs/1905.09274) describes the motivation and context for this API.
13+
The main motivation can be quoted from section 3.3 of that paper:
14+
15+
> (Property1) **Application message retrieval partitioning.** Client nodes must be able to download all of the messages relevant to the applications they use [...], without needing to downloading any messages for other applications.
16+
17+
> (Property2) **Application message retrieval completeness.** When client nodes download messages relevant to the applications they use [...], they must be able to verify that the messages they received are the complete set of messages relevant to their applications, for specific
18+
blocks, and that there are no omitted messages.
19+
20+
21+
22+
The main data structure that enables above properties is called a Namespaced Merkle Tree (NMT), an ordered binary Merkle tree where:
23+
1. each node in the tree includes the range of namespaces of the messages in all descendants of each node
24+
2. leaves in the tree are ordered by the namespace identifiers of the leaf messages
25+
26+
A more formal description can be found the [specification](https://github.com/celestiaorg/celestia-specs/blob/de5f4f74f56922e9fa735ef79d9e6e6492a2bad1/specs/data_structures.md#namespace-merkle-tree).
27+
An implementation can be found in [this repository](https://github.com/celestiaorg/nmt).
28+
29+
This ADR basically describes version of the [`GetWithProof`](https://github.com/celestiaorg/nmt/blob/ddcc72040149c115f83b2199eafabf3127ae12ac/nmt.go#L193-L196) of the NMT that leverages the fact that IPFS uses content addressing and that we have implemented an [IPLD plugin](https://github.com/celestiaorg/celestia-core/tree/37502aac69d755c189df37642b87327772f4ac2a/p2p/ipld) for an NMT.
30+
31+
**Note**: The APIs defined here will be particularly relevant for Optimistic Rollup (full) nodes that want to download their Rollup's data (see [celestiaorg/optimint#48](https://github.com/celestiaorg/optimint/issues/48)).
32+
Another potential use-case of this API could be for so-called [light validator nodes](https://github.com/celestiaorg/celestia-specs/blob/master/specs/node_types.md#node-type-definitions) that want to download and replay the state-relevant portion of the block data, i.e. transactions with [reserved namespace IDs](https://github.com/celestiaorg/celestia-specs/blob/master/specs/consensus.md#reserved-namespace-ids).
33+
34+
## Alternative Approaches
35+
36+
The approach described below will rely on IPFS' block exchange protocol (bitswap) and DHT; IPFS's implementation will be used as a black box to find peers that can serve the requested data.
37+
This will likely be much slower than it potentially could be and for a first implementation we intentionally do not incorporate the optimizations that we could.
38+
39+
We briefly mention potential optimizations for the future here:
40+
- Use of [graphsync](https://github.com/ipld/specs/blob/5d3a3485c5fe2863d613cd9d6e18f96e5e568d16/block-layer/graphsync/graphsync.md) instead of [bitswap](https://docs.ipfs.io/concepts/bitswap/) and use of [IPLD selectors](https://github.com/ipld/specs/blob/5d3a3485c5fe2863d613cd9d6e18f96e5e568d16/design/history/exploration-reports/2018.10-selectors-design-goals.md)
41+
- expose an API to be able to download application specific data by namespace (including proofs) with the minimal number of round-trips (e.g. finding nodes that expose an RPC endpoint like [`GetWithProof`](https://github.com/celestiaorg/nmt/blob/ddcc72040149c115f83b2199eafabf3127ae12ac/nmt.go#L193-L196))
42+
43+
## Decision
44+
45+
Most discussions on this particular API happened either on calls or on other non-documented way.
46+
We only describe the decision in this section.
47+
48+
We decide to implement the simplest approach first.
49+
We first describe the protocol informally here and explain why this fulfils (Property1) and (Property2) in the [Context](#context) section above.
50+
51+
In the case that leaves with the requested namespace exist, this basically boils down to the following: traverse the tree starting from the root until finding first leaf (start) with the namespace in question, then directly request and download all leaves coming after the start until the namespace changes to a greater than the requested one again.
52+
In the case that no leaves with the requested namespace exist in the tree, we traverse the tree to find the leaf in the position in the tree where the namespace would have been and download the neighbouring leaves.
53+
54+
This is pretty much what the [`ProveNamespace`](https://github.com/celestiaorg/nmt/blob/ddcc72040149c115f83b2199eafabf3127ae12ac/nmt.go#L132-L146) method does but using IPFS we can simply locate and then request the leaves, and the corresponding inner proof nodes will automatically be downloaded on the way, too.
55+
56+
## Detailed Design
57+
58+
We define one function that returns all shares of a block belonging to a requested namespace and block (via the block's data availability header).
59+
See [`ComputeShares`](https://github.com/celestiaorg/celestia-core/blob/1a08b430a8885654b6e020ac588b1080e999170c/types/block.go#L1371) for reference how encode the block data into namespace shares.
60+
61+
```go
62+
// RetrieveShares returns all raw data (raw shares) of the passed-in
63+
// namespace ID nID and included in the block with the DataAvailabilityHeader dah.
64+
func RetrieveShares(
65+
ctx context.Context,
66+
nID namespace.ID,
67+
dah *types.DataAvailabilityHeader,
68+
api coreiface.CoreAPI,
69+
) ([][]byte, error) {
70+
// 1. Find the row root(s) that contains the namespace ID nID
71+
// 2. Traverse the corresponding tree(s) according to the
72+
// above informally described algorithm and get the corresponding
73+
// leaves (if any)
74+
// 3. Return all (raw) shares corresponding to the nID
75+
}
76+
77+
```
78+
79+
Additionally, we define two functions that use the first one above to:
80+
1. return all the parsed (non-padding) data with [reserved namespace IDs](https://github.com/celestiaorg/celestia-specs/blob/de5f4f74f56922e9fa735ef79d9e6e6492a2bad1/specs/consensus.md#reserved-namespace-ids): transactions, intermediate state roots, evidence.
81+
2. return all application specific blobs (shares) belonging to one namespace ID parsed as a slice of Messages ([specification](https://github.com/celestiaorg/celestia-specs/blob/de5f4f74f56922e9fa735ef79d9e6e6492a2bad1/specs/data_structures.md#message) and [code](https://github.com/celestiaorg/celestia-core/blob/1a08b430a8885654b6e020ac588b1080e999170c/types/block.go#L1336)).
82+
83+
The latter two methods might require moving or exporting a few currently unexported functions that (currently) live in [share_merging.go](https://github.com/celestiaorg/celestia-core/blob/1a08b430a8885654b6e020ac588b1080e999170c/types/share_merging.go#L57-L76) and could be implemented in a separate pull request.
84+
85+
```go
86+
// RetrieveStateRelevantMessages returns all state-relevant transactions
87+
// (transactions, intermediate state roots, and evidence) included in a block
88+
// with the DataAvailabilityHeader dah.
89+
func RetrieveStateRelevantMessages(
90+
ctx context.Context,
91+
nID namespace.ID,
92+
dah *types.DataAvailabilityHeader,
93+
api coreiface.CoreAPI,
94+
) (Txs, IntermediateStateRoots, EvidenceData, error) {
95+
// like RetrieveShares but for all reserved namespaces
96+
// additionally the shares are parsed (merged) into the
97+
// corresponding types in the return arguments
98+
}
99+
```
100+
101+
```go
102+
// RetrieveMessages returns all Messages of the passed-in
103+
// namespace ID and included in the block with the DataAvailabilityHeader dah.
104+
func RetrieveMessages(
105+
ctx context.Context,
106+
nID namespace.ID,
107+
dah *types.DataAvailabilityHeader,
108+
api coreiface.CoreAPI,
109+
) (Messages, error) {
110+
// like RetrieveShares but this additionally parsed the shares
111+
// into the Messages type
112+
}
113+
```
114+
115+
## Status
116+
117+
Proposed
118+
119+
## Consequences
120+
121+
This API will most likely be used by Rollups too.
122+
We should document it properly and move it together with relevant parts from ADR 002 into a separate go-package.
123+
124+
### Positive
125+
126+
- easy to implement with the existing code (see [ADR 002](https://github.com/celestiaorg/celestia-core/blob/47d6c965704e102ae877b2f4e10aeab782d9c648/docs/adr/adr-002-ipld-da-sampling.md#detailed-design))
127+
- resilient data retrieval via a p2p network
128+
- dependence on a mature and well-tested code-base with a large and welcoming community
129+
130+
### Negative
131+
132+
- with IPFS, we inherit the fact that potentially a lot of round-trips are done until the data is fully downloaded; in other words: this could end up way slower than potentially possible
133+
- anyone interacting with that API needs to run an IPFS node
134+
135+
### Neutral
136+
137+
- optimizations can happen incrementally once we have an initial working version
138+
139+
## References
140+
141+
We've linked to all references throughout the ADR.

‎docs/celestia-architecture/adr-004-mvp-light-client.md

+292
Large diffs are not rendered by default.

‎docs/celestia-architecture/adr-005-decouple-blockid-and-partsetheader.md

+47
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# ADR 005: Decouple the PartSetHeader from the BlockID
2+
3+
## Changelog
4+
5+
- 2021-08-01: Initial Draft
6+
7+
## Context
8+
9+
Celestia has multiple commits to the block data via the `DataHash` and the `PartSetHeader` in the `BlockID`. As stated in the [#184](https://github.com/celestiaorg/lazyledger-core/issues/184), we no longer need the `PartSetHeader` for this additional commitment to the block's data. However, we are still planning to use the `PartSetHeader` for block propagation during consensus in the short-medium term. This means that we will remove the `PartSetHeader` from as many places as possible, but keep it in the `Proposal` struct.
10+
11+
## Alternative Approaches
12+
13+
It’s worth noting that there are proposed changes to remove the `PartSetHeader` entirely, and instead use the already existing commitment to block data, the `DataAvailabilityHeader`, to propagate blocks in parallel during consensus. Discussions regarding the detailed differences entailed in each approach are documented in that ADR's PR. The current direction that is described in this ADR is significantly more conservative in its approach, but it is not strictly an alternative to other designs. This is because other designs would also require removal of the `PartSethHeader`, which is a project in and of itself due to the `BlockID` widespread usage throughout tendermint and the bugs that pop up when attempting to remove it.
14+
15+
## Decision
16+
17+
While we build other better designs to experiment with, we will continue to implement the design specified here as it is not orthogonal. https://github.com/celestiaorg/lazyledger-core/pull/434#issuecomment-869158788
18+
19+
## Detailed Design
20+
21+
- [X] Decouple the BlockID and the PartSetHeader [#441](https://github.com/celestiaorg/lazyledger-core/pull/441)
22+
- [ ] Remove the BlockID from every possible struct other than the `Proposal`
23+
- [X] Stop signing over the `PartSetHeader` while voting [#457](https://github.com/celestiaorg/lazyledger-core/pull/457)
24+
- [X] Remove the `PartSetHeader` from the Header [#457](https://github.com/celestiaorg/lazyledger-core/pull/457)
25+
- [X] Remove the `PartSetHeader` from `VoteSetBits`, `VoteSetMaj23`, and `state.State` [#479](https://github.com/celestiaorg/lazyledger-core/pull/479)
26+
- [ ] Remove the `PartSetHeader` from other structs
27+
28+
29+
## Status
30+
31+
Proposed
32+
33+
### Positive
34+
35+
- Conservative and easy to implement
36+
- Acts as a stepping stone for other better designs
37+
- Allows us to use 64kb sized chunks, which are well tested
38+
39+
### Negative
40+
41+
- Not an ideal design as we still have to include an extra commitment to the block's data in the proposal
42+
43+
## References
44+
45+
Alternative ADR [#434](https://github.com/celestiaorg/lazyledger-core/pull/434)
46+
Alternative implementation [#427](https://github.com/celestiaorg/lazyledger-core/pull/427) and [#443](https://github.com/celestiaorg/lazyledger-core/pull/443)
47+
[Comment](https://github.com/celestiaorg/lazyledger-core/pull/434#issuecomment-869158788) that summarizes decision

‎docs/celestia-architecture/adr-006-row-propagation.md

+202
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,202 @@
1+
# ADR 006: Consensus Block Gossiping with Rows
2+
3+
## Changelog
4+
* 24.06.2021 - Initial description
5+
* 07.07.2021 - More important details were added
6+
* 18.08.2021 - Mention alternative approaches briefly
7+
8+
## Context
9+
It's a long story of relations between Celestia, Tendermint, and consensus block gossiping. Celestia's team discussed
10+
multiple ideas, several ADRs were made, and nothing yet was finalized. This ADR is another attempt to bring valuable
11+
changes into block gossiping and hopefully successful.
12+
13+
Currently, we inherit the following from Tendermint. Our codebase relies on the blocks Parts notion. Each Part is a
14+
piece of an entire serialized block. Those Parts are gossiped between nodes in consensus and committed with
15+
`PartSetHeader` containing a Merkle Root of the Parts. However, Parts gossiping wasn't designed for Celestia blocks.
16+
17+
Celestia comes with a different block representation from Tendermint. It lays out Blocks as a table of data shares,
18+
where Rows or Columns can be and should be gossiped instead of Parts, keeping only one system-wide commitment to data.
19+
20+
## Alternative Approaches
21+
### ["nah it works just don't touch it"](https://ahseeit.com//king-include/uploads/2020/11/121269295_375504380484919_2997236194077828589_n-6586327691.jpg) approach
22+
23+
It turns out that we could fully treat the Tendermint consensus as a black box, keeping two data commitments: one for
24+
consensus with `PartSetHeader` and another for the world outside the consensus with `DAHeader`.
25+
26+
#### Pros
27+
* Less work
28+
29+
### Others
30+
* get rid of the PartsHeader from BlockID without changing block propagation at all (see [ADR 005](https://github.com/celestiaorg/celestia-core/blob/58a3901827afbf97852d807de34a2b66f93e0eb6/docs/lazy-adr/adr-005-decouple-blockid-and-partsetheader.md#adr-005-decouple-the-partsetheader-from-the-blockid))
31+
* change block propagation to fixed-sized chunks but based on the ODS instead of how Parts are built currently (for this we have empirical evidence of how it performs in practice)
32+
* send the block as a whole (only works with smaller blocks)
33+
* block propagation-based on sending the header and Tx-IDs and then requesting the Tx/Messages that are missing from the local mempool of a node on demand
34+
35+
#### Cons
36+
* Pulls two data commitments to Celestia's specs
37+
* Brings ambiguity to data integrity verification
38+
* Controversial from software design perspective
39+
* Brings DOSing vector for big Blocks. Every Block would need to be represented in two formats in RAM
40+
* Wastes more resources on building and verifying additional
41+
42+
## Decision
43+
The decision is to still treat Tendermint's consensus as a black box, but with few amendments to gossiping mechanism:
44+
* Introduce `RowSet` that mimics `PartSet`.
45+
46+
`RowSet` is a helper structure that wraps DAHeader and tracks received Rows with their integrity against DAHeader and
47+
tells its user when the block is complete and/or can be recovered. Mostly it is a helper and is not a high-level
48+
concept.
49+
* Replace `PartSet` with `RowSet` within consensus.
50+
* Keep `DAHeader` in `Proposal`
51+
* Remove `PartSetHeader` from `Proposal`
52+
53+
The changes above are required to implement the decision. At later point, other changes listed below are
54+
likely to be implemented as a clean-up:
55+
* Entirely removing `PartSetHeader`, as redundant data commitment
56+
* Removing `PartSet`
57+
* Relying on `DAHeader` instead of `PartSetHeader`
58+
59+
## Detailed Design
60+
The detailed design section demonstrates the design and supporting changes package by package. Fortunately, the
61+
design does not affect any public API and changes are solely internal.
62+
63+
### `types`
64+
#### RowSet and Row
65+
First and essential part is to implement `RowSet` and `Row`, fully mimicking semantics of `PartSet` and `Part` to
66+
decrease the number of required changes. Below, implementation semantics are presented:
67+
68+
```go
69+
// Row represents a blob of multiple ExtendedDataSquare shares.
70+
// Practically, it is half of an extended row, as other half can be recomputed.
71+
type Row struct {
72+
// Index is an top-to-bottom index of a Row in ExtendedDataSquare.
73+
// NOTE: Row Index is unnecessary, as we can determine it's Index by hash from DAHeader. However, Index removal
74+
// would bring more changes to Consensus Reactor with arguable pros of less bandwidth usage.
75+
Index int
76+
// The actual share blob.
77+
Data []byte
78+
}
79+
80+
// NewRow creates new Row from flattened shares and index.
81+
func NewRow(idx int, row [][]byte) *Row
82+
83+
// RowSet wraps DAHeader and tracks added Rows with their integrity against DAHeader.
84+
// It allows user to check whenever rsmt2d.ExtendedDataSquare can be recovered.
85+
//
86+
// RowSet tracks the whole ExtendedDataSquare, Where Q0 is the original block data:
87+
// ---- ----
88+
// | Q0 || Q1 |
89+
// ---- ----
90+
// | Q2 || Q3 |
91+
// ---- ----
92+
//
93+
// But its AddRow and GetRow methods accepts and returns only half of the Rows - Q0 and Q2. Q1 and Q3 are recomputed.
94+
// ----
95+
// | Q0 |
96+
// ----
97+
// | Q2 |
98+
// ----
99+
//
100+
type RowSet interface {
101+
// NOTE: The RowSet is defined as an interface for simplicity. In practice it should be a struct with one and only
102+
// implementation.
103+
104+
// AddRow adds a Row to the set. It returns true with nil error in case Row was successfully added.
105+
// The logic for Row is:
106+
// * Check if it was already added
107+
// * Verify its size corresponds to DAHeader
108+
// * Extend it with erasure coding and compute a NMT Root over it
109+
// * Verify that the NMT Root corresponds to DAHeader Root under its Index
110+
// * Finally add it to set and mark as added.
111+
//
112+
AddRow(*Row) (bool, error)
113+
114+
// GetRow return of a Row by its index, if exist.
115+
GetRow(i int) *Row
116+
117+
// Square checks if enough rows were added and returns recomputed ExtendedDataSquare if enough
118+
Square() (*rsmt2d.ExtendedDataSquare, error)
119+
120+
// other helper methods are omitted
121+
}
122+
123+
// NewRowSet creates full RowSet from rsmt2d.ExtendedDataSquare to gossip it to others through GetRow.
124+
func NewRowSet(eds *rsmt2d.ExtendedDataSquare) *RowSet
125+
126+
// NewRowSetFromHeader creates empty RowSet from a DAHeader to receive and verify gossiped Rows against the DAHeader
127+
// with AddRow.
128+
func NewRowSetFromHeader(dah *ipld.DataAvailabilityHeader) *RowSet
129+
```
130+
131+
#### Vote
132+
`Vote` should include a commitment to data. Previously, it relied on `PartSetHeader` in `BlockId`, instead it relies on
133+
added `DAHeader`. Protobuf schema is updated accordingly.
134+
135+
#### Proposal
136+
`Proposal` is extended with `NumOriginalDataShares`. This is an optimization that
137+
helps Validators to populate Header without counting original data shares themselves from a block received form a
138+
Proposer. Potentially, that introduce a vulnerability by which a Proposer can send wrong value, leaving the populated
139+
Header of Validators wrong. This part of the decision is optional.
140+
141+
### `consenSUS`
142+
#### Reactor
143+
##### Messages
144+
The decision affects two messages on consensus reactor:
145+
* `BlockPartMessage` -> `BlockRowMessage`
146+
* Instead of `Part` it carries `Row` defined above.
147+
* `NewValidBlockMessage`
148+
* Instead of `PartSetHeader` it carries `DAHeader`
149+
* `BitArray` of `RowSet` instead of `PartSet`
150+
Protobuf schema for both is updated accordingly.
151+
152+
##### PeerRoundState
153+
`PeerRoundState` tracks state of each known peer in a round, specifically what commitment it has for a Block and what
154+
chunks peer holds. The decision changes it to track `DAHeader` instead of `PartSetHeader`, along with `BitArray` of
155+
`RowSet` instead of `PartSet`.
156+
157+
##### BlockCatchup
158+
The Reactor helps its peers to catchup if they go out of sync. Instead of sending random `Part` it now sends random
159+
`Row` by `BlockRowMessage`. Unfortunately, that requires the Reactor to load whole Block from store. As an optimization,
160+
an ability to load Row only from the store could be introduced at later point.
161+
162+
#### State
163+
##### RoundState
164+
The RoundState keeps Proposal, Valid and Lock Block's data. Along with an entire Block and its Parts, the RoundState
165+
also keeps Rows using `RowSet`. At later point, `PartSet` that tracks part can be removed.
166+
167+
##### Proposal Stage
168+
Previously, the State in proposal stage waited for all Parts to assemble the entire Block. Instead, the State waits for
169+
the half of all Rows from a proposer and/or peers to recompute the Block's data and notifies them back that no more
170+
needs to be sent. Also, through Rows, only minimally required amount of information is gossiped. Everything else to
171+
assemble the full Block is collected from own chain State and Proposal.
172+
173+
## Status
174+
Proposed
175+
176+
## Consequences
177+
### Positive
178+
* Hardening of consensus gossiping with erasure coding
179+
* Blocks exceeding the size limit are immediately rejected on Proposal, without the need to download an entire Block.
180+
* More control over Row message size during consensus, comparing to Part message, as last part of the block always has
181+
unpredictable size. `DAHeader`, on the other hand, allows knowing precisely the size of Row messages.
182+
* Less bandwidth usage
183+
* Only required Block's data is gossiped.
184+
* Merkle proofs of Parts are not sent on the wire
185+
* Only one system-wide block data commitment schema
186+
* We don't abandon the work we were doing for months and taking profits out of it
187+
* PR [#287](https://github.com/celestiaorg/lazyledger-core/pull/287)
188+
* PR [#312](https://github.com/celestiaorg/lazyledger-core/pull/312)
189+
* PR [#427](https://github.com/celestiaorg/lazyledger-core/pull/427)
190+
* and merged others
191+
192+
### Negative
193+
* We invest some more time(~1.5 weeks).
194+
* Most of the work is done. Only few changes left in the implementation along with peer reviews.
195+
196+
### Neutral
197+
* Rows vs Parts on the wire
198+
* Previously, parts were propagated with max size of 64KiB. Let's now take a Row of the largest 128x128 block in
199+
comparison. The actual data size in such a case for the Row would be 128x256(shares_per_row*share_size)=32KiB, which
200+
is exactly two times smaller than a Part.
201+
* Gossiped chunks are no longer constant size. Instead, their size is proportional to the size of Block's data.
202+
* Another step back from original Tendermint's codebases

‎docs/celestia-architecture/adr-template.md

+72
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# ADR {ADR-NUMBER}: {TITLE}
2+
3+
## Changelog
4+
5+
- {date}: {changelog}
6+
7+
## Context
8+
9+
> This section contains all the context one needs to understand the current state, and why there is a problem. It should be as succinct as possible and introduce the high level idea behind the solution.
10+
11+
## Alternative Approaches
12+
13+
> This section contains information around alternative options that are considered before making a decision. It should contain a explanation on why the alternative approach(es) were not chosen.
14+
15+
## Decision
16+
17+
> This section records the decision that was made.
18+
> It is best to record as much info as possible from the discussion that happened. This aids in not having to go back to the Pull Request to get the needed information.
19+
20+
## Detailed Design
21+
22+
> This section does not need to be filled in at the start of the ADR, but must be completed prior to the merging of the implementation.
23+
>
24+
> Here are some common questions that get answered as part of the detailed design:
25+
>
26+
> - What are the user requirements?
27+
>
28+
> - What systems will be affected?
29+
>
30+
> - What new data structures are needed, what data structures will be changed?
31+
>
32+
> - What new APIs will be needed, what APIs will be changed?
33+
>
34+
> - What are the efficiency considerations (time/space)?
35+
>
36+
> - What are the expected access patterns (load/throughput)?
37+
>
38+
> - Are there any logging, monitoring or observability needs?
39+
>
40+
> - Are there any security considerations?
41+
>
42+
> - Are there any privacy considerations?
43+
>
44+
> - How will the changes be tested?
45+
>
46+
> - If the change is large, how will the changes be broken up for ease of review?
47+
>
48+
> - Will these changes require a breaking (major) release?
49+
>
50+
> - Does this change require coordination with the Celestia fork of the SDK or celestia-app?
51+
52+
## Status
53+
54+
> A decision may be "proposed" if it hasn't been agreed upon yet, or "accepted" once it is agreed upon. Once the ADR has been implemented mark the ADR as "implemented". If a later ADR changes or reverses a decision, it may be marked as "deprecated" or "superseded" with a reference to its replacement.
55+
56+
{Deprecated|Proposed|Accepted|Declined}
57+
58+
## Consequences
59+
60+
> This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones.
61+
62+
### Positive
63+
64+
### Negative
65+
66+
### Neutral
67+
68+
## References
69+
70+
> Are there any relevant PR comments, issues that led up to this, or articles referenced for why we made the given design choice? If so link them here!
71+
72+
- {reference link}

‎docs/celestia-architecture/assets/user-request.png

388 KB
Loading

‎docs/celestia-architecture/celestia-logo.png

40 KB
Loading

‎docs/celestia-architecture/img/extended_square.png

13.5 KB
Loading

0 commit comments

Comments
 (0)
Please sign in to comment.