A Polkadot Postmortem — 24.05.2021
By: Bastian Köcher
TL;DR: On 24 May 2021, Polkadot nodes failed with an out of memory (OOM) error on block 5,202,216. This block contained an on-chain solution to the validator election, which is normally computed off-chain and only takes place on-chain if no off-chain solution is submitted. Due to the large number of nominators, the election overflowed the memory allocated in the Wasm environment.
While an update was being prepared to fix the issue, validators were asked to temporarily downgrade their node software to a previous version that includes a native (non-Wasm) version of the runtime. The native version is not constrained by the Wasm memory allocator. The network recovered after an hour and ten minutes of downtime.