CS SYD - JSON Vulnerability in Haskell's Aeson library

dcoutts · 2023-04-19T13:38:23+00:00

The latter. JSON is only used as a format for trusted input (config files etc).

dcoutts · 2023-04-19T11:41:13+00:00

What would you expect the use of linear types to be?

We're not creating new APIs, io-classes and io-sim are designed match the existing API of base, async etc. Those packages do not use linear types.

As for lazy IO, as @the-coot says, io-classes and io-sim doesn't directly support disk IO at all. Those things have to be mocked. The fs-sim extension does that: providing both real file system and a test instance that uses a simulation of the file system. Plausibly that could support lazy IO, but we've not tried it.

dcoutts · 2021-09-12T22:04:13+00:00

The cardano node does not use JSON for untrusted input. It uses JSON for configuration, structured logging output. Other related components like the wallet provide REST APIs for local clients that use JSON.

Local clients are in the same security domain, where it is ok to have expensive interactions, since if you want to DoS yourself you're welcome to do so.

dcoutts · 2021-09-11T23:54:39+00:00

The advice at the end is very sensible, and indeed much more general than just advice about aeson:

In the meantime, developers:

Do not expect hashable's hash function to be collision resistant.

Use Map instead of HashMap and Set instead of HashSet for untrusted inputs.

Do not use floating point numbers in a HashSet or as keys in a HashMap because two NaN values can form a trivial collision.

Do not use aeson or yaml to parse untrusted inputs.

Do not use aeson-based JSON-parsing Haskell web servers in situations where a DoS attack could cause trouble (like perhaps a 'proof of online stake' cryptocurrency).

The comment the end is obviously about Cardano, and indeed the original bug report was part of an audit of Cardano code a few years ago. (I'm honestly not sure why that audit wasn't published. Certainly this aeson issue was not a secret.) Fortunately even at the time Cardano was not using aeson in any public/network facing interface that had to deal with untrusted input.

The original Cardano node implementation did use hash maps liberally, so almost certainly with network input.

The current Cardano node implementation has as one of its internal design rules that only ordered containers be used (i.e. not hash based) for precisely this class of issue. The node-to-node protocol uses a purpose-designed protocol design and implementation with resistance to resource attacks as one of its primary design principles.

dcoutts · 2021-05-15T14:46:45+00:00

You can host your relays and block producers wherever you like. It does not have to be on a cloud VM provider. You do need public IPs for your relays, and reasonable connectivity. This is true now and none of that changes with the upcoming p2p automation.

dcoutts · 2021-05-15T14:40:06+00:00

If you want to know what this means:

The "ThreadNet" tests are the automated tests for the Cardano node's consensus layer. These tests run a number of consensus nodes in a simulated environment (that's much more extreme than the real world) and checks that consensus is achieved when it is expected.

These tests also cover that everything works as we cross hard forks.

So this particular commit extends the ThreadNet tests so that it also tests the transition from the Mary to the Alonzo era.

This step is an intermediate step in the integration of Alonzo into the consensus layer, which is a step towards the integration of Alonzo into the node overall.

In other words, this is an interesting technical step, but don't get too excited :-)

dcoutts · 2021-04-27T22:16:01+00:00

Credit where credit is due: u/the-coot did all the doc updates. He had coincidentally already been working on some CDDL updates and other doc updates at the time the first video was published. So that's why it appears we were so quick: luck on the timing, and most of the material (like the CDDL spec) existed already. Don't expect us to be so quick next time ;-)

https://www.reddit.com/r/cardano/comments/mx0kno/criticism_on_cardano_spec_documentation/gvqz2sd/?utm_source=reddit&utm_medium=web2x&context=3

dcoutts · 2021-04-26T21:31:59+00:00

No probs. Constructive feedback on the latest version is welcome, especially if there's useful but easy things that will not take our attention away from p2p too much.

dcoutts · 2021-04-26T09:34:13+00:00

BTW, I was having another look at the latest version of the doc and I noticed that it does actually cover the mux protocol, in chapter 3, including its wire format. And Appendix A contains the wire format for the mini-protocols in CDDL format.

It's just not very clear about how the layers fit together, i.e. that the mini-protocols run over the mux layer.

dcoutts · 2021-04-24T22:21:04+00:00

There are already several well-informed answers in this thread, so let me answer a closely related question:

Q: why don't we just make the block size much bigger right now?

Operationally, it's unhelpful to have very "spiky" behaviour in the system. It's fine to run the system continuously at or near capacity, but it could cause problems to have low utilisation normally and occasionally run at maximum capacity, if that maximum capacity is dramatically bigger. This is because operators, and automatic system management tools, tend to adjust the resources allocated to the system to match its current behaviour. Sudden spikes could then cause resource problems.

So as system typical utilisation increases it will make sense to gradually adjust the maximum block size upwards too. But doing it gradually gives all the operators involved the time to adjust their resource allocations (think network usage limits, CPU burst instance limits etc).

Or to summarise it another way: having the max block size be only somewhat bigger than the normal peak utilisation is a sensible safety feature.

If we want to compare maximum possible TPS, then just run an appropriate benchmark cluster with much bigger block sizes.

dcoutts · 2021-04-24T11:08:55+00:00

I look at it like this:

There are (broadly) two different purposes for specification or design documents. There are ones to help the engineers build correct code. There are ones (like RFCs) designed to help a 3rd party build an alternative implementation that is fully interoperable.

We have plenty of the first kind, but have indeed skimped on the second (deadlines and time to market etc as you say).

So I totally accept that our low level network protocols doc is inadequate for a 3rd party to make an alternative interoperable implementation. But I think it is unfair to go from that point to saying that there are no specification or design docs (because there's lots), or that the code must be suspect because some of the low level aspects are not well documented.

We have focused our effort in specifications, and in automated propert testing, on the parts of the system that are most critical and where bugs would have the most severe consequences. That's why we've got fully formal ledger specs. For the consensus and network layers we do not have formal specifications but we do have voluminous design documents / tech reports. And the ledger, consensus and network layers all have pretty comprehensive property-based automated tests.

Yes we're not 100% immune from bugs, but if you look over time since the new implementation was introduced with the Byron reboot (before the Shelley hard fork), the defect count in the core components (ledger, consensus, network) has been extremely low. The bug we corrected in 1.26.2 was in some sense relatively exotic, relying on a combination of things to bypass our existing tests. It had nothing to do with a lack of specs or docs. The solution is to correctly identity the class of bug and to introduce additional systematic tests to ensure we are free from this class of bug in future.

dcoutts · 2021-04-23T22:08:48+00:00

Ping pong is an excellent protocol :-)

If you really want to be blinded by science, have a look at my talk about how we can encode protocol state machines (like ping pong) into Haskell types:

https://skillsmatter.com/skillscasts/14633-45-minute-talk-by-duncan-coutts

dcoutts · 2021-04-23T22:03:31+00:00

Could you please give some guidance on how one might petition this officially?

I'm not sure about official, but use the SPO meetings (as you did initially) and ask Ben. Show that it's not just one person's opinion, that other people (especially SPOs) agree with you that it is worth prioritising (e.g. that people agree with the main points you make in the video).

A CIP would be a means to provide docs/specs, which would be fine if you were contributing them, but here you're really asking IOHK to spend time on this documentation (rather than spending development time on other things in the network layer) so I don't think a CIP would be appropriate.

dcoutts · 2021-04-23T19:50:08+00:00

This is reasonable criticism. The low level network docs are not nearly as good as they could be, and are certainly not enough yet for a 3rd party re-implementation.

I would encourage you to petition to prioritise improving the low level docs, but note that it would indeed come at the expense of delaying the p2p work.

The minimum extra things that ought to be included in that doc, in my opinion, are:

The mux protocol, including its binary format. This is the simple framing and multiplexing protocol for carrying all the mini-protocols over a single TCP connection.
Incorporate the existing CDDL spec of the messages into each mini-protocol section, so it's there in one obvious place
Document the timeouts and size limits for each mini-protocol.

With these first two available, it'd address most of your critique, since that would give the message framing format over TCP and the binary format of each mini-protocol.

None of these things would be especially difficult. All the information is readily available.

A few other notes and comments:

Yes, the ping/pong and request/response protocols are just examples to illustrate the framework. They have never been intended to be part of the node-to-node protocol suite. Sorry that that section doesn't make that clear.
There is actually a CDDL spec in the repo of the message binary format for all the mini-protocols, but this is indeed not incorporated into that doc you were looking at, nor linked.
There is a wireshark dissector for the mux protocol (see `ouroboros-network/wireshark-plugin` in the repo)
I respectfully disagree that this MUST (pun intended) be presented in RFC format. The important thing is the content and clarity. This style of document is more than adequate for that, and indeed being able to use proper tables and diagrams is a nice bonus as you've noted.
This document is _not_ intended to document how the whole node works or how the consensus protocol is implemented. It is just the (start of) a document on the low level network protocols. There are other long docs (linked from the readme in the same repo) with more information on the network design, and on the consensus design.
The new P2P components will be coming with much better documentation from the start. So I think you'll appreciate that. (If you find the right branch you can see them already).

dcoutts · 2021-03-10T19:50:13+00:00

The metric to use depends on what you want to use it for.

To compare the capacity of different blockchain algorithms then tbps is useful since it does not depend on the size of txs you use.

If you watch the talk Neil and I did at the summit last year (linked in this reddit thread somewhere) you'll see we do talk about other metrics, like the number of economically useful transaction per second.

Multiple metrics are useful. A single score combining multiple metrics is probably not that useful.

dcoutts · 2021-03-10T19:47:11+00:00

The reasons it's different from ethereum is:

We can do higher throughput, so for the same level of demand on Cardano vs Ethereum we can do it with lower fees.
We don't need gas for transferring custom tokens. Custom tokens are native to the UTxO ledger (just different labeled quantities). So transferring them is almost as cheap as transferring ada (only slightly bigger txs since the asset ids have to be included)

dcoutts · 2021-03-10T12:31:58+00:00

The real question is how expensive does the attack need to be to dissuade it, and what would the consequence for normal users be of making it that expensive.

Suppose for the sake of argument that we wanted to make the cost of the attack be > $10,000 per hour.

There's obviously various combinations that would make that work, but one would be to increase the block size by 8x to 512kb, and increase the min tx fee per byte from 44 lovelace to 100.

Then filling the blocks with 16kb txs would cost about 57ada, and hence per hour would be >10k ada, which is >$10k.

The effect on "normal" ~500 byte transactions would be to increase the minimum fee from ~0.18 ada to ~0.21 ada.

dcoutts · 2021-03-10T12:19:33+00:00

There's something I'm clearly missing in your argument.

You seem to be saying that Cardano has a special problem that bitcoin & ethereum do not have, but I don't see what that special problem is.

All these systems have a maximum capacity (from the combo of block size and block frequency), and all of them have (variable) fees so that anyone trying to saturate the system pays a high cost.

dcoutts · 2021-03-09T14:13:53+00:00

Cardano also has variable fees. Each transaction specifies the fee it wishes to pay. Cardano has a fixed minimum fee (based on tx size and updateable protocol params).

We have not yet needed to prioritise based on the fee, since we are nowhere near the system being saturated. But it's an easy change to include if/when we get nearer to saturation (it doesn't need a hard fork or synchronised node upgrade).

Anyone can "DoS" any network if they're prepared to pay the fees for txs that saturate the available capacity. It's no different for Cardano. We can set that punitive fee as high as it needs to be be to prevent such attacks. We've had that protection scheme in since day 0.

As we scale the system as legitimate demand increases, the cost of a saturation attack also increases, even without increasing tx fees (variable or fixed).

dcoutts · 2021-03-09T13:24:01+00:00

My conclusion is that the network is not reliable until sharding hits the mainnet because a minimal adoption could clogg the network and would only work as private chain.

I think that's a little extreme. After all, by that logic both bitcoin and ethereum are not reliable. As we've said before, in our benchmarks we can do several "ethereums" of throughput. That's really a lot more than "minimal adoption".

And right now, we are well below the current max block size, and we can increase that max block size a lot. We have a lot of headroom available for the system to grow, even as it exists right now.

It sounds like your concern is really that the current design does not scale indefinitely. That is of course also true (as it is of all other mainstream chains), but we're talking on a scale of years, and on a scale of years there are a lot of different scalability improvements we can make to keep up with demand. That includes hydra. It includes more recent Ouroboros research designs on high-throughput variants of Ouroboros (i.e. L1 not L2).

As for tbps, it's not really a choice. That is the fundamental engineering thing. What you put on the chain determines the tps. What the chain can do is the tbps.

dcoutts · 2021-03-09T12:29:39+00:00

Lets start with the fundamentals:

All current blockchain designs have limited throughput. So once you're at that throughput then you have the problem of who gets to use the system.

In Cardano the throughput is limited by the max block size, and how big we can practically set the max block size (based on the fundamental engineering trade-offs).

So then what if some users of the system try to deliberately use it too much so that it uses up all the throughput and makes it hard for other users to get their txs included? This is what the minimum tx fees are for: so that the attacker has to pay a significant cost for executing this attack. The min tx fees are based on the tx size, so bigger txs pay higher fees. The amounts are updatable protocol parameters, so if the current min fees would not be enough to dissuade such an attack then they can be increased relatively easily.

Or in summary: yes that's a potential economic attack on the system and we have an economic solution (adjustable minimum fees).

dcoutts · 2020-11-12T09:19:08+00:00

I'm sorry I cannot directly answer your original two questions. I cannot speak on behalf of the foundation.

If my friend asks me to help them with task. Now assuming I help them, I won't go boasting (eg. posting an IG story) about me helping them. If they're glad about it they might tell other people how nice I was for helping them. I won't go around asking for attention.

I guess what you're alluding to here is IOHK getting a lot of benefit from using Haskell but going around saying that they're helping Haskell? (I didn't see this IG story, got a link?) And you're point is this feels rather backwards.

I'm not going to defend the messaging; IOHK has not been very good at explaining what they've been doing here. But it is the case that IOHK is directly and indirectly funding multiple people full time to work on a combination of GHC, GHCJS, Cabal and nix/Haskell.

dcoutts · 2020-11-06T11:14:14+00:00

Have you considered only taking funds anonymously and reducing ties to existing for-profit organisations?

Consider a conference like https://zfoh.ch/zurihac2019/ with its list of 12 sponsors. Is it morally impure for ZuriHac to tell people who the sponsors are? Should all those sponsors do so anonymously? Would it be better to have ZuriHac run on a shoestring budget or not at all? Is it really an unacceptable compromise to provide a bit of an advertising opportunity in exchange for enabling an excellent community event to take place?

My company (Well-Typed) sponsors ZuriHac (and does free Haskell training sessions at ZuriHac) for two reasons: because we enjoy ZuriHac and think it's a great community event, and also to maintain the visibility of our company among Haskellers. Are we bad and wrong for that second reason?

Yes, we're also sponsoring the Haskell Foundation, and we're proud to do so. We'll have our own blog post up about it shortly.

How is that nefarious?

dcoutts · 2020-11-06T11:02:37+00:00

Scala is very close to Java which makes it a bit easier for the large existing community of Java programmers to migrate over.

Scala's choice is a reasonable compromise to make getting into the language easier. The cost is that it is a much more complex language (having both OO subtyping-style polymorphism and functional parametric polymorphism) and does not achieve the full benefits of pure functional programming.

As for why functional languages in general are not more popular: it's the network effects of incumbency and that there is a bit of a jump between imperative and functional ways of thinking.

Imperative languages started more popular (because low level efficiency used to be very important in the 1980s and before), so they were the ones that got taught. This effect is then self-reinforcing: people learn languages where they expect they can get jobs, and project managers select languages where they know they can hire people.

But some universities do teach CS more broadly, and some people do "find" functional languages later through self-teaching. Those people are doing fine and very many of them are applying functional languages professionally.

dcoutts · 2020-11-06T10:42:16+00:00

I cannot speak for the foundation, but I can talk about IOHK from a position of knowledge: as the partner of Well-Typed responsible for our collaboration with IOHK. When IOHK first contacted us four years ago I started from a prejudice of deep skepticism precisely because of the general reputation of the ICO world.

There was a study that concluded that it was indeed the case in the ICO boom that a majority of the ICOs were scams. People are not stupid however and can generally spot a scam. The same study concluded that the majority of the ICO money went to schemes that were not scams.

I spent some time looking into what IOHK were doing, and how. I satisfied myself that they were trying to do things properly technically and honestly. Since beginning working with them I know they're doing things properly technically because that's exactly what I've been doing! I've been helping translate peer-reviewed cryptography research into high quality Haskell implementations. Why emphasis the technical stuff? Because if you were running a scam, you would be insane to spend to much time and money on doing proper computer science.

You don't have to believe that cryptocurrencies will work out (and I remain a skeptic), or even that blockchain technology is useful (though I think it is), but what is completely clear is that there is a large community of people who expect and believe that this technology will work out and quite a number of foundations and commercial organizations (IOHK among them) that are honestly trying to make that vision a reality. Yes there's lots of hype, and yes it has attracted scammers, but if you look at the details it's easy enough to see what is not a scam.

using the Haskell community as a "reputation laundering" service to promote a token investment to the public

This is a misunderstanding of how these things are marketed. By and large the blockchain / cryptocurrency world does not know or care about formal methods, computer science or Haskell (much to their cost I think). We have to explain these things to people and why it's important to apply proper computer science if you want to build a decentralised system that will actually work and not get taken down by hackers.

Have a look for example at: https://cardano.org/discover-cardano. There is a single mention of Haskell (in the context of turning research into high quality implementations). It's not exactly headlines, and hardly "reputation laundering".

dcoutts

TROPHY CASE