CCIP in production: what we learned running cross-chain messaging

Xora Labs Engineering  ·  12 min read

Sending a cross-chain message is a two-hour tutorial. Running one as production infrastructure, where a stuck message means somebody's funds are in neither place, is a different discipline entirely. These are the things we wish someone had told us before the first deployment.

The mental model that helps

Treat a cross-chain message as an asynchronous job submitted to a queue you do not control, running on a machine you cannot restart, with a receipt you must handle exactly once. Every design mistake we have seen came from someone reasoning about it as a function call that happens to be slow.

In particular: the message will land, but you cannot say when. It may land after your contract state has moved on. It may land after your front end has told the user the operation failed. Your receiving contract has to be correct for all of those orderings, not just the happy one.

Lanes are configuration, and configuration drifts

A lane is the directed pair of chains you are allowed to send between, with its own rate limits, its own supported tokens and its own enable/disable state. Two things follow from that which teams routinely miss.

• Lanes are directional. Enabling A→B tells you nothing about B→A, and half-configured pairs are a very common cause of “it works in one direction” bug reports.
• Rate limits are enforced per lane, and they refill over time. A burst that succeeds in testing can fail in production simply because someone else drained the capacity first.

We keep lane configuration in the same repository as the contracts, generated from a single source of truth, and we diff live on-chain configuration against it on a schedule. Configuration that only lives in a deployment script is configuration that will silently diverge.

The failure modes that actually happen

In roughly descending order of how often they have bitten us:

• Receiver runs out of gas. The gas limit is specified by the sender, on the source chain, for execution on a destination chain whose costs it cannot observe. Under-provision and the message is delivered but the handler reverts.
• Token pool liquidity is exhausted. Burn-and-mint hides this; lock-and-release does not. If the destination pool is empty, transfers queue behind whoever refills it.
• The receiver reverts on valid input. A require() that was reasonable on the source chain is not always reasonable when the message arrives forty minutes later.
• Rate limit rejection. Correct behaviour, surprising timing, and almost always discovered in production rather than staging.

Write the receiver so that a message which cannot be processed is stored, not reverted. A revert throws away the only copy of the instruction you were sent.

This is the single highest-value change we make to client code. Accept the message, validate it, and if it cannot be executed now, persist it with a status and emit an event. Retry from an operational path. A reverted message is a support ticket; a stored one is a queue entry.

Idempotency is not optional

Assume, as a design premise, that your handler can be invoked more than once for the same logical instruction, whether through a manual re-execution, an operational retry, or an integration mistake on the sending side. Key on the message identifier, record it before you act on it, and make repeat delivery a cheap no-op.

The failure this prevents is the expensive kind: crediting the same deposit twice. It costs almost nothing to build in on day one and is difficult to retrofit once state has accumulated.

What to monitor before you launch

Anything that only alerts on reverts will miss the failure that matters most, which is a message that was accepted and then simply never arrived. We instrument four things:

• Send-to-execute latency, as a distribution rather than an average. The tail is the story.
• In-flight count and age. Alert on the oldest unexecuted message, not the total.
• Remaining rate-limit capacity per lane, so you see exhaustion coming instead of discovering it.
• Balance and reconciliation on both sides. Source debits should equal destination credits plus in-flight. Divergence is the earliest signal that something is genuinely wrong.

What we do differently now

We build the operational path before the happy path: a way to inspect stuck messages, a way to replay them, and a reconciliation job that runs whether or not anyone is watching. We also insist on a kill switch on the receiving side that pauses processing without rejecting messages, so a bad deployment degrades into a queue rather than a loss.

None of this is exotic. It is ordinary distributed-systems practice, applied to a setting where the retry costs gas and the queue is somebody else's.

Working on something like this? Get in touch.