Base identifies sequencer bug behind two-day mainnet outages
Coinbase-backed Layer-2 network Base has confirmed the same bug in its sequencer block-builder logic caused two separate block-production outages on June 25–26. The first outage lasted roughly 116 minutes; the second lasted about 20 minutes. Base says all funds remained safe throughout both incidents.
What happened
- A transaction that failed during execution left stale journal state inside the block builder. That stale state included accounts and storage slots touched by the failed transaction.
- When a subsequent valid transaction executed, the block builder used the wrong journal state and charged gas incorrectly, producing a block with an invalid state transition.
- Because other nodes could not accept the invalid block, the network’s sequencer stopped producing new L2 blocks and block production halted.
User impact
- Users could not get new transactions included on-chain while sequencing was halted. Transactions queued in the mempool; the pool later grew beyond capacity, causing new eth_sendRawTransaction calls to return errors during the outage windows.
- Sequencer and validator progress were also affected because nodes could not move past the invalid block until sequencing resumed.
Fixes and root causes
- Base applied a sequencer patch to ensure journal state updates correctly after a failed transaction, which addresses the primary bug.
- During recovery engineers discovered a second issue: a race condition in the engine reset feature that prevented sequencers from catching up after restarts. That contributed to the incident returning the next day. This problem affected sequencers rather than validator nodes and slowed recovery.
- Base resumed sequencing on June 25 and advised ecosystem node operators to restart their nodes if they remained stuck.
Postmortem and next steps
- Base published an official postmortem and shared findings with OP chains for feedback. The team plans to strengthen protocol fuzz testing and load testing to expose unusual transaction patterns that can surface hidden bugs.
- Additional measures include improved monitoring and operational checks, and plans to add graceful recovery to base-consensus so validator nodes can continue syncing after similar failures.
- Base says it has named the bug, released a patch, and listed the tests it will improve.
Context
- The outage occurred during a busy week for Base, which is also progressing its Beryl upgrade that introduces the B20 token standard and shortens the standard Base-to-Ethereum withdrawal period from seven days to five days.
Bottom line: Base has identified and patched the sequencer bug that caused both outages, confirmed funds were safe, and outlined testing and operational upgrades aimed at preventing similar incidents in future.
Read more AI-generated news on: undefined/news
