Post-mortem Report for Mainnet Upgrade v8.1.0 Background: T | TRUSTWALLET SUPPORT ❇️
Post-mortem Report for Mainnet Upgrade v8.1.0
Background: The Zilliqa protocol underwent a mainnet upgrade on Aug 30, 2021. This upgrade introduced important features to enable ZilBridge such as:
- Merkle Patricia tree data structure for contract storage
- Contract proof API needed for PolyNet relayers
It also included several stability, maintenance and bug fixes such as:
- Several memory clean up improvements
- Improve network stability for JSON RPC API service
- Improve PoW submission handling
- Node syncing improvement
The upgrade took a few hours and once the upgrade was completed, miners had started to join the network and started processing transactions, some exchanges like Binance had also opened up deposit and withdrawals.
Issue reported: At this point, one of the community devs from the Duck team reached out with an observation with their NFT token minting contract. They noticed that there were two transactions, one with TXID starting with 0xef and another starting with 0x02 which had minted the same token_id 1351. The first transaction was done before the upgrade had started, while the second one after the upgrade.
For some reasons, the transaction and therefore the state change that happened before the upgrade was ignored and the network after the upgrade wasn’t aware that the token_id 1351 had already been sold and hence after the upgrade, it was resold to a new buyer.
Early investigation and first action taken: As soon as we were informed about this, we started to look into the code of the Duck NFT minting contract. The contract code looked correct. After investigating a bit more, we suspected that the mainnet post-upgrade had started with a wrong global state and as a result the transaction before the upgrade even though it did happen, was considered to be non-existent from the network’s perspective. Since the global state that the nodes started with post-upgrade was somehow corrupted, we suspected the impact to be beyond the Duck NFT contract.
This took us a few hours and by that time 4,478 transactions had been processed by the network post-upgrade. Many of these transactions had been processed with the corrupt initial global state. In order to reduce the impact of the corrupted global state, we decided to disable RPC endpoints that allow users to send transactions. By doing so, we limited any future transaction to be processed under a wrong global state.
Investigating the root cause: After making sure that we had restricted the impact by disabling transactions, we started to investigate a bit deeper into what had gone wrong. We found that when the network was upgraded to v8.1.0 at block number 1,394,088, the final global state after the upgrade was not the same as the global state before the upgrade had started. In other words, the global state after block 1,394,088 was not the same as the state before block 1,394,089.
The main reason behind this was that the state changes for the last 88 blocks (i.e., for all the blocks processed in the last epoch, each epoch is characterised by 100 blocks) were not fully taken into account for the global state post-upgrade. The state changes for these 88 blocks are stored in what’s called a state delta as these 88 blocks had not formed a full 100-blocks epoch. Consider state delta as a temporary change in the global state to be effectuated at the end of the epoch. Nodes keep these state deltas separately until the epoch is fully complete. At the end of the epoch (i.e., every 100 blocks), the state delta is then merged with the global state.
To summarize, the issue was that while all the merged state deltas were considered part of the global state (i.e., all transactions until block 1,394,000), transactions made in the last 88 blocks such as the DUCK NFT minting transaction were ignored. As a result, even though those transactions were processed by the network, the state after the upgrade did not consider the state change that they created. It was as if those transactions never occurred.
