Reorgs, backfills and the indexer problems nobody warns you about

Xora Labs Engineering  ·  11 min read

The first version of every indexer is a loop that reads blocks and writes rows, and it works beautifully right up until it does not. The failures are not exotic. They are the same four problems, in the same order, on every project.

Problem one: the chain is not append-only

Blocks you have already processed can be replaced. Shallow reorgs are routine on most chains and deeper ones happen under stress. If your indexer treats a block as final the moment it reads it, your database will confidently contain events that no longer happened.

The fix is not to wait for finality on everything, because that makes the data uselessly late. It is to make your writes reversible for the unfinalised window. We keep a rolling buffer of recent blocks with the state changes each one produced, and on reorg we unwind to the fork point and replay. Beyond the finality horizon the buffer is discarded and the data is treated as permanent.

• Store the block hash with every row, not just the number. Number alone cannot detect a reorg.
• Make unwind a first-class operation you test deliberately, not a path exercised for the first time in production.
• Different chains have very different finality semantics. Treating a probabilistic chain and a fast-finality chain identically is a decision, so make it consciously.

Problem two: backfill and live tail collide

You backfill history from block zero while also following the head, and eventually the two meet. Almost every naive implementation either double-writes the overlap or leaves a gap at the join, and both are discovered weeks later when someone queries a range nobody had queried before.

Design the join before you start the backfill. Discovering it at block 40,000 means starting again.

What works is making every write idempotent and keyed deterministically, typically by block hash, transaction index and log index, so overlap is harmless by construction. Then the backfill and the live tail can run concurrently, meet anywhere, and the result is identical either way.

Problem three: the schema will change

You will need to add a field, fix a decoding bug, or handle a contract upgrade that changed an event signature. If reindexing means taking the service down for two days, you will avoid it, and you will ship worse data instead.

We keep raw decoded logs separately from derived tables. Raw data is expensive to re-fetch and cheap to keep; derived tables are cheap to rebuild and are what actually changes. A schema fix becomes a rebuild from local raw data in minutes rather than a re-scan of the chain over days.

Problem four: silence is not success

An indexer that has quietly stopped looks exactly like an indexer on a quiet chain. Both produce no new rows. The alert you need is not “did it error” but “how far behind head are we, right now”, measured continuously and alerted on the number rather than on the absence of exceptions.

• Lag in blocks and in seconds, because blocks alone mislead when block time varies.
• Time since the last successful write, which catches a stalled loop that is not erroring.
• Reorg depth observed, which is an early warning that the chain is under stress.
• A periodic consistency check against the node: for a sample of past blocks, does what we stored still match what the chain says?

What we build for clients

All of the above, before any business logic is written. The schema is yours. We build to the shape your product needs rather than making you adapt to a generic one. But the reorg handling, the idempotent keys, the raw/derived split and the lag alerting are not negotiable, because they are what separates an indexer from a script that has not failed yet.

Working on something like this? Get in touch.