The Soft Fork Bulletin Issue #3
Screaming at a Wall: The Challenge of Blockchain State Growth
Mental model
A blockchain is a replicated deterministic state machine.
To grasp that concept, picture a spreadsheet with rows listing accounts and columns listing their balances. That's the state. Now imagine one of those accounts makes a payment to another. That's the transition. Keep a record of that payment — the output — and update the relevant account balances to create a new state. Then go back to the top, and do the whole thing again, and again, and again, ad infinitum. You have a state machine.
Now make sure you have rules in place so that a given input applied to a given state always creates the same subsequent state and output. In this case, for instance, an account has to have sufficient balance to make a payment, and a payment decreases the payer account's balance and increases the payee account's balance by the exact same amount every time. Make sure those rules can't be violated. If you replay a transaction — you start with the same state and apply the same transition — you get the same result. You have made your state machine deterministic.
Now keep copies of that deterministic state machine on different physical computers. Make sure those computers are networked, that they communicate to share state and input data, and that they follow the same rules for transitions. In this example, they start with the same spreadsheet of accounts and balances, they receive the same request to make a payment, and they apply the same function to update balances accordingly. Use some sort of consensus algorithm to make sure those computers agree on state and transition data, and that they apply transitions in lockstep. You have replicated your deterministic state machine.
At this point, you're running a little blockchain in your mind. Congratulations! Obviously, you still have a lot to figure out — you need to come up with a consensus algorithm to coordinate replication; you need to define deterministic functions to ensure consistent state changes and output; you need a method to store the output in some sort of history archive — but don't worry about all that right now. Right now, just run your mental machine for a bit, and then turn your attention to one specific part of the model: the state.
State growth
Your blockchain, like all blockchains, begins with some initial state. Call it the genesis state, or the genesis block, or the genesis ledger. All different names for the same thing. In your case, it starts with just two accounts. Simple! But since you want more people to use your blockchain, you include some method for them to create new accounts. Every time they do, every computer that replicates the state machine adds a row to their copy of the spreadsheet. Two rows becomes a hundred, a thousand, a million, a hundred million. Your blockchain is succeeding! The state keeps getting bigger!
Let's say you've convinced me to participate in your blockchain. I like what you're building, I want it to succeed, I believe decentralization is important for censorship resistance. I dedicate this computer, the one I'm writing this newsletter on, to your replicated state machine. It's using that nifty consensus protocol I asked you not to worry about to coordinate with the other computers, and it's storing state in RAM so it can access it fast enough to keep up with all of the transitions.
This computer has 16 GB of RAM. How much memory does your state take up? When it just stores a hundred accounts, not much. 16 GB of RAM is enough to handle it. But what about when it stores a billion? At that point, the state is too big for RAM, so the computer starts writing the excess data to disk, and there's actually a limit to how quickly a computer can do that.
According to Wikipedia, which as you know is 100% accurate 100% of the time, "Input/output operations per second (IOPS, pronounced eye-ops) is an input/output performance measurement used to characterize computer storage devices." While there are different measurements of IOPS, I'm going to avoid the weeds to keep things moving here. You can, for the purposes of this newsletter, just think of IOPS as the performance of a storage device under typical usage. IOPS, pronounced eye-ops!
Based on what I believe is the correct spec, the IOPS for this computer's hard drive is 230,000. The computer can't read from or write to it any faster than that. At this point, more users are joining your blockchain, and they're creating more and more new accounts, and this computer is applying transitions to update state, and to do that, it's relying on input/output operations because it's out of RAM. Will it be able to keep up? What happens if it can't?
The death trajectory
Those are, more or less, the questions Péter Szilágyi, the lead Geth developer, asks in a recent talk called "Ethereum in numbers: Where TPS meets physics."
The Ethereum state is a bit more sophisticated than our mental model: it is stored in a large data structure called a Merkle Patricia Trie, and according to the docs, it "holds not only all accounts and balances, but a machine state, which can change from block to block according to a pre-defined set of rules, and which can execute arbitrary machine code." I won't get into the gory details of that data structure here, but, as Szilágyi says in the talk, "all the accounts are laid out flat, and we have a cryptographic tree structure on top to just try to prove that everything is correct." So while the Ethereum state holds a more robust data set than our mental model blockchain, it's also structured to compress that data to reduce storage requirements.
However, as Szilágyi explains, that data structure does not solve the problem of state growth. Ethereum currently has about 174 million accounts, and for an optimized node, the state takes up about 770 GB. That's already big enough to exceed the RAM of most computers, and it continues to grow by about 50-100 GB per year, so increasingly, Ethereum nodes have to push state to disk. According to Szilágyi, a single transaction currently requires 420 input/output operations to do that. The suggested spec for Ethereum node hardware — SSD NVMe over PCIe 3 — has an IOPS of 360,000. Divide that by 420 (the input/output operations required for a transaction), and you get the maximum number of transactions per second a node that is pushing state to disk can handle: 857.
Ethereum is configured to process way fewer transactions per second than that — somewhere around 15 — so you may think: "Nice! Plenty of headroom!" The problem is that as state grows, so too does the number of input/output operations required to write transactions to disk. The tree gets bigger, and it takes a computer more effort to traverse and append new entries to it. As Szilágyi says, "If you take any of the Ethereum networks…and you just stick a constant transaction throughput on it, the state will grow…and the problem is that the more that number grows, the higher RAM requirements we have, and the higher RAM requirements we have, the more and more data we flush to disk, the more IOPS we generate, and essentially we're hitting that brick wall." If state growth hits the limits of hard drive capability, and computers can't keep up with incoming transactions, your replicated state machine may just...stop. Final word from Szilágyi: "Ethereum — and all its forks — is kind of like on a potential death trajectory currently."
What can you do?
The death trajectory is not unique to Ethereum, nor is it unique to forks of Ethereum like Avalanche C-Chain and Binance Smart Chain. It's a problem for your mental model blockchain; it's a problem for blockchains in general. It's also a known problem, one protocol developers throughout the industry are thinking about.
It's come up a few times lately in my life because Stellar developers are in the process of building a smart contracts platform called Soroban, and as part of that work, they've started having open discussions about how to prevent state growth. The current Stellar state resembles our mental model blockchain — it's a ledger with a few simple entry types including accounts, balances, and orders to buy and sell, and it's pretty lean compared to Ethereum — but smart contracts require more data, so their introduction will likely accelerate ledger growth.
The technical proposal for a fee structure for Stellar smart contracts attempts to address that potential acceleration, and it includes a whole section on ledger size reduction that lays out three core design principles for a solution:
Ledger space is a shared public resource, so policies should be set to ensure fair use.
The cost of using ledger space should converge towards market rate over time
In particular, creating spam ledger entries should cost market rate over the long term.
Abandoned entries should not cost anything to network participants over the long term.
And that proposal — like similar proposals in other blockchain ecosystems — grapples with a difficult fact: to prevent unbounded state growth, you have to find a way to remove "abandoned entries" from the ledger.
Why is that fact difficult? Imagine you have a balance of a million USDC. It's held in an account, and that account is stored as a ledger entry. You don't want to trade or make payments at the moment, you just want to hodl. Maybe your local currency is highly inflationary, and you're keeping that balance as a hedge. You don't touch your account for a couple of years, but then you want to buy a house, and you decide to use that money to do it. When you try to access your account, you find…it's been deleted. Your money is gone. You've lost a million dollars.
Granted, that's a hypothetical situation, but what a terrible outcome! Unacceptable! Surely there's a way to prevent state growth without burning users' money! There must be, right? The answer to that question is… probably yes?
That fee structure proposal, for instance, kicked off an ongoing discussion on the Stellar developer mailing list about how to archive state rather than simply deleting it. Ledger entries that aren't touched for a certain amount of time would still expire, but there would be a mechanism for motivated users to resurrect them. It would probably require the creation of a new kind of node that would store archived entries and provide proofs of existence to allow for their reactivation. But because those proofs can be slower than the blockchain itself — if it takes an hour to revive an account, that's probably fine! — those nodes wouldn't have to keep up with the pace of state transitions, so they wouldn't run into the same problems with input/output operation speed limits. That could work!
Solana, meanwhile, seems to have implemented a proposal to charge rent for accounts and, if I'm reading it correctly, accounts that don't pay rent are, in fact, deleted: "Under the preceding design, it is NOT possible to have accounts that linger, never get touched, and never have to pay rent." That said, they do seem sensitive to the downsides — rent is automatically deducted, and accounts with a balance equivalent to two years of rent payments are exempt — so it sounds like you couldn't actually lose a million dollars due to inactivity. You could lose something, just not a million dollars. Curious to see how that goes!
Ethereum devs are also working on a solution. You're probably familiar with the next upcoming technical change that the network will undergo: the Merge, which is scheduled for mid-September, will switch the consensus mechanism from proof of work to proof of stake. But do you know about the subsequent "urges"?
The Surge will implement sharding
The Verge will introduce a more efficient data structure called a Verkle tree
The Purge will introduce a form of state expiration
And the Splurge will add "miscellaneous but important extras!"
All those plans are pretty ornate, and I won't try to explain them in detail here and now. The first two should help slow state growth — the Surge allows nodes to store partial state, so data requirements decrease; the Verge makes it so that nodes need to store state to produce new blocks, but not to verify them — but neither actually stops it. It's not until the Purge that a plan kicks in to put a boundary on state size. That plan also involves state expiration, and, like the Stellar proposal, it looks like it does include a method to resurrect inactive state. But again: it's still an idea, not something that's been executed. How soon will the Purge happen? Hard to say. However…
"We are kind of running out of time," says Szilágyi in that aforementioned talk. It's hard to say when, but eventually, all blockchains that allow state to grow unchecked will arrive at the limits imposed by hardware and, to quote Szilágyi one last time, "It's a dangerous place to be at to arrive at the brick wall without a solution." And while there are a few possible solutions on the table, none are battle-tested, none are tried-and-true. Will any of these approaches hold up technically? Will ecosystems agree to adopt them? Will users understand them? Will people lose money? If they do, will they jump ship?
For me, part of what makes working in blockchain exciting is that there's still a lot to figure out, and those questions…they represent an interesting challenge: the challenge of blockchain state growth.
We track TPS at https://chainspect.app/. You are right! Ethereum real-time TPS is 14.77 tx/s, and max recorded Ethereum TPS is 57.91 tx/s.