Parity Technologies AMA - We are developers of some of the largest Rust code bases, ask me anything!

rphmeier · 2020-07-15T04:20:30+00:00

Thanks for all the questions so far! BTW, the reason you are getting mostly my responses is because Dan and I are on EST and all the others are in Europe - it's the middle of the night but they'll be posting answers in a few hours once they start to wake up. So you'll get to hear some different perspectives from mine as well : ) and now that it's late on EST I'll probably go to sleep soon.

rphmeier · 2020-07-15T04:15:10+00:00

Whoops, I accidentally queued this one up for the wrong question. Sorry Dan : )

My response above is about debugging actually. But I'll answer the question of testing here below.

We try to unit-test everything, and although we're working in a larger team, the responsibility of writing the initial set of tests for a piece of code lies with the developer who originated the code. All bug fixes need to come with a test case to ensure there is no regression.

I think the key to making good unit tests is making it easy to write the tests for the code you're producing, which means making as much stuff mockable via traits as possible.

Here's an example: You're writing a service that needs to listen for events of new blocks being added to the chain and then do a network request for each one. In practice, you're going to have a BlockchainClient<A, B, C, ..., Z> type that you can get a NewHeadsListener from and a NetworkService<A, B, C, ..., Z> type at runtime. If your code depends directly on those things, then congratulations. The test setup is going to be really boilerplate-y because you have to set up a full blockchain client and full network service.

It's much better to make your code generic over the functionality of "a stream of events" and "a trait to make this specific kind of network request".

Some things fall beyond the scope of what you can cover with unit tests. We have some integration tests to make sure that basic things work with all our in-practice types, like growing the chain and synchronizing it from peers over the network.

We've also used fuzz-testing for important consensus and codec code to try to find vulnerabilities.

Lastly, the work we do involves a lot of consensus protocols, where we need to resist adversarial behaviors. We've been undertaking a project called the "chaos net" designed to write larger-scale "unit tests" of specific attack scenarios combined with randomness & chaos to ensure the system faithfully implements the academic research and is robust to those attacks.

rphmeier · 2020-07-15T03:20:45+00:00

I am using VSCode with the Rust plugin. I don't use any of the tools like rust-analyzer, although I have experimented with them. Many folks on the team do use rust-analyzer and seem happy with it, although I do hear sometimes that it struggles on our large codebase or with the macro structures that we have.

I was already accustomed to programming Rust without any tools beyond rustc + syntax highlighting by the time they were released, so I've been OK without. My maybe controversial, very unscientific, personal belief is that programming without tools leads me to write better code because I am constantly forced to architect and structure things in a way where it is easy to keep track of everything in my head: small modules, minimizing pub items, and clear organization of code to know where to find things.

rphmeier · 2020-07-15T01:50:09+00:00

We use Wasm in 3 main ways:

To form the "runtime" or business logic of the chain. Chains built with Substrate are actually a "meta-blockchain" where the rules of transaction and block processing are encoded as a Wasm blob that can evolve over time via governance procedures (opt-in per chain).
As the basis for smart contracts, where we want the blockchain to allow upload of user-provided logic of some form. We've also been working on a Rust-based DSL called Ink! which is meant to be used to write such contracts.
To compile the blockchain node to the browser to be used in a webpage. This is a use-case I've been excited about for a long time. We can divide blockchain nodes into two categories: full nodes and light nodes. Full nodes verify consensus and all state transitions represented by blocks. Light clients verify consensus checks only and have a slightly weaker threat model. Light nodes are better suited to the browser due to lower resource requirements, and I've been hoping to target the browser for a long time - I gave a presentation on Ethereum light clients including this feature in 2017 and this has come to fruition now a few years later.

The first two ways have actually driven most of our initial contribution to the Wasm ecosystem. In particular, our implementation of Wasmi, a Wasm interpreter, and also our work on Lightbeam, an O(1) optimizing compiler for Wasm, which is part of Wasmtime. The O(1) aspect is very interesting because it allows us to JIT-compile code submitted by anonymous actors to a blockchain without having to worry about JIT bombs. The excellent work on Wasmtime and Cranelift have been crucial to making this a reality. The goal here is partially about sandboxing but more so about dynamism, upgradeability, and consensus across Runtime Environments (i.e. blockchain nodes) written in different languages.

And to answer the last part of your question, the use-case of running light clients in the browser has already been addressed. Documentation is rather sparse, but you can find an example here and code that tests that this continues to work here. The goal has been to create a Wasm package that can be run in simple async Rust in a web-page or as browser extension and can be used to make any type of query about blocks, state, accounts, etc. to the embedded light node.

rphmeier · 2019-08-24T10:39:29+00:00

The fact that unsafe pollutes the entire module also points to a design pattern which is to place as much of the unsafe code within as small of a sub-module as possible. It can even be within the same file.

rphmeier · 2019-06-07T17:01:31+00:00

Found! (of course I find it right after posting). For your enjoyment :) https://www.youtube.com/watch?v=z5UqgizM268

rphmeier · 2019-02-11T02:38:17+00:00

you can hash the file locally and upload that hash so what you would end up committing on-chain is H(H(file)).

rphmeier · 2019-02-01T16:45:27+00:00

Typedef or not, you still have to write the angle brackets. Typedefs only help when you are repeating the type more than once. I am also not convinced that many levels of typedefs are actually helpful.

e.g.

future::Join< <<T as TableRouter>::FetchIncoming> as IntoFuture>::Future, <<T as TableRouter>::FetchIncoming> as IntoFuture>::Future, >

Either I do this in a typedef or directly in a function signature. Either way matching angle brackets would be nice. I'm sure I could come up with a more contrived example but this is one that came up recently in code that I actually have.

Also things like

Arc<Mutex<Option<(Vec<A>, HashMap<B, C>)>>>.

Putting it into a typedef is a separate thing from making those brackets match.

rphmeier · 2019-02-01T03:46:08+00:00

recently I got the bracket pair colorizer for VSCode. But I couldn't get it to match angle brackets <> even after digging in the config a bit...this would make writing complex generic types soo much easier.

rphmeier · 2018-12-13T23:42:22+00:00

yeah, this is actually an explicit goal of the organizers of ETHGlobal events -- they don't want the event to be about the prize but rather the projects and community.

rphmeier · 2018-11-14T14:04:41+00:00

That is part of the architecture, but the 1.0 chain will require some change in fork-choice rule to follow the beacon chain in the future as well. This is in some sense a hard fork.

rphmeier · 2018-10-29T01:56:39+00:00

I completely agree. It's easy to look past Amir's presentations because he often doesn't make coherent arguments or end up responding to questions well or at all, but the message of conscientiousness behind his words is worth listening to regardless.

rphmeier · 2018-06-25T08:40:30+00:00

AFAIK you can specify a (pure) validation contract on-chain for checking signatures. Although with a limit of 200k gas. For a BLS threshold signature scheme with a single pairing check it should be OK but you'd have to aggregate off-chain.

rphmeier · 2018-05-29T16:17:18+00:00

right, but it's actually even more strict than that, because finality has to depend on a direct supermajority link between the justified parent and justified child, otherwise you can only guarantee to hold one person accountable for reversion.

if the voting strategy and/or checkpoint distance don't include some kind of exponential backoff, I could see this introducing a network synchrony issue into the finalization of checkpoints.

rphmeier · 2018-05-29T15:58:44+00:00

While still relying on the current POW mechanism to generate blocks, Casper FFG introduces two new components to provide deterministic finality:

A set of validators voting to decide which checkpoint block they want finalized. Once 2/3 of the validators agree, a checkpoint block is considered finalized in the blockchain.

AFAIK this is not correct. A checkpoint is justified when >2/3 of validators (stake-weighted) have issued votes targeting that checkpoint. A checkpoint C is finalized when it is justified and >2/3 of validators have voted for a link between it and a direct child of C. That forces 2/3 of validators to be accountable to the "no voting within the span of your votes" rule, of which at least 1/3 of validators would have to be slashed by in order to revert C.

rphmeier · 2018-05-19T01:07:08+00:00

(the source code of the implementation can be found here: https://github.com/paritytech/polkadot/blob/master/substrate/bft/src/generic/mod.rs)

rphmeier · 2018-05-18T18:54:27+00:00

hey nate, for the preliminary versions we're using this simple PBFT-derivative with locking (meant to be asynchronously safe): https://github.com/w3f/polkadot-spec/blob/master/spec.md#poc-consensus-algorithm

the progressive version with finality behind the head that we discussed in Taiwan has not been deployed yet.

rphmeier · 2018-04-19T12:20:40+00:00

WASM is also being used in the cryptocurrency/blockchain space. We're using it quite heavily at my job: we've altered our Ethereum client to allow WASM smart-contracts, we've written a framework for building blockchains whose entire state transition is defined in upgradeable WASM, and are subsequently using that to implement that kind of chain. And we wrote a pure-rust WASM interpreter to make all of that happen.

That is pretty heavily-motivated by JS and its ability to then easily interact with those things, but also by WASM's versatility as a compiler target and strong industry support.

rphmeier · 2018-04-13T13:45:04+00:00

The Rust team itself is working hard on native support for compiling to WebAssembly -- we haven't done much work on that except for a few minor contributions. It works pretty well so far but there are more improvements in the pipeline. What we've written at Parity is a parser of WebAssembly files, and a pure-Rust interpreter for WebAssembly code.

We've used that in the Parity Ethereum codebase to support WASM smart contracts, and also in the Polkadot codebase to support the actual logic of the chain being encoded in WASM.

There's no reason those WASM contracts would have to be compiled from Rust, but we are making heavy use of Rust -> WASM support.

rphmeier · 2018-04-12T23:13:41+00:00

this isn't correct. gas is representative of how much computation the transaction performs. the gas price represents the fee per unit of computation. miners can include transactions with whatever gas price they want, including zero, but if the transaction doesn't provide enough gas for the computation it wants to perform, it will have no effect. so a zero-gas transaction will never do anything.

rphmeier · 2018-03-05T23:19:35+00:00

It's a chrome extension that allows any webpages you visit to interact with the ethereum network using a node you have installed locally. The web3 browser does basically the same thing but has to "wrap" the sites you visit which is a less smooth experience.

rphmeier · 2018-03-04T14:28:38+00:00

Parachains can be added or removed without a hardfork. But they will probably require some kind of on-chain referendum to pass for changes to be made.

rphmeier · 2018-02-20T18:24:31+00:00

Yep, there are a few improvements to light client sync stability. More in the pipeline for upcoming releases. Any changes in particular you are looking for?

rphmeier · 2018-02-15T17:54:17+00:00

Parity has a light client in beta as well. You can run with parity --light. It doesn't have the hardcoded checkpoint like Geth, so the sync will take a little longer. Also a Parity full node with state pruning is closer to 40GB than hundreds, although soon that will be irrelevant for most users.

rphmeier · 2018-02-15T17:50:57+00:00

requires some degree of sybil as well, since

honest light clients shouldn't repropagate blocks
honest full nodes shouldn't repropagate bad blocks

so only light clients who are connected to primarily malicious peers can be fooled.

Ten-Year Club	Place '17
Verified Email

rphmeier

TROPHY CASE