Switcheo TradeHub Incident Report (8th Feb 2021)

Switcheo TradeHub suffered a chain halt on 8th Feb 2021 at 10:06PM SGT, which was unrelated to separate issue found on the preceding day. This is a post-mortem report for the incident. In preparation for allowing trading of futures contracts, oracles for the BTCUSD and ETHUSD index was created on 16th Dec 2020. This meant that validators began submitting votes for prices of these assets every second. However, after the V1.12.0 upgrade in early February, some validators stopped running the oracle service on their subaccounts. Because a supermajority was not present, no oracle results were resolved for an extended period of time, while the oracle votes remained in the chain state. This implementation is intended to give a buffer period for slower oracles to join in each voting round. The oracle module then automatically purges votes that are beyond a certain age at the end of each block. However, it only performs this purge whenever a new valid result is resolved.