Designing for a region outage without rebuilding everything

Regional outages are rare and consequential. The standard advice, run active-active across regions, is correct and also expensive, complicated and frequently disproportionate to the actual risk a given organisation carries.

Between “single region, hope for the best” and “full active-active” there is a wide middle ground that delivers most of the practical benefit for a fraction of the cost. Most teams never explore it because the conversation jumps straight to the hardest option and then stalls.

Decide what you are actually promising

Two numbers drive everything, and they should be decided by whoever owns the business consequence rather than by whoever will implement it.

Recovery time objective is how long you can be down. Recovery point objective is how much data you can afford to lose.

The answers differ enormously by system. A marketing site can be down for hours and lose nothing, because there is nothing to lose. A payment system may tolerate minutes and no data loss at all. An internal reporting tool might comfortably tolerate a day.

Applying the same target to everything is the most common and most expensive mistake in this area. Write the numbers down per system, and let them drive the design rather than the reverse.

The levels, in increasing cost

Level zero: single region, backups elsewhere. Everything runs in one region; backups are stored in another region or another provider. A regional outage means downtime for its duration, and a regional loss means rebuilding from backup. Recovery time is hours to days.

This is genuinely adequate for a great many internal systems, and it is dramatically better than backups stored in the same region, which is where most people actually are.

Level one: multiple zones within a region. Instances spread across availability zones behind a load balancer, database with a standby in another zone. This covers the far more common failure of a single data centre or zone, costs relatively little, and is well supported by managed services. A full regional outage still takes you down.

For most applications this is the correct place to stop. Zone failures are considerably more frequent than regional ones, and this handles them with automatic failover.

Level two: warm standby in a second region. Infrastructure defined in code so the environment can be recreated, data replicated continuously, and a scaled-down or stopped environment ready to bring up. Recovery time is minutes to an hour, mostly spent on DNS and scaling. Cost is modest, because the standby runs small or not at all.

This is the sweet spot for systems that matter but do not justify active-active.

Level three: active-active across regions. Both regions serve traffic at all times. Recovery is near-instant. Cost is at least double, and the engineering problem is genuinely difficult: writes in both regions mean conflict resolution, replication lag becomes user-visible, and split-brain scenarios need designing for.

Choose this when the numbers you wrote down demand it, not in anticipation.

The parts people forget

Whichever level you choose, the same items are routinely missed and each can prevent a recovery from working.

DNS with a low enough time-to-live. If your records have a day-long TTL, failover takes a day regardless of how ready the standby is. Lower it in advance; doing it during an incident does not help, because the old value is already cached.

Infrastructure defined in code. If the second region has to be built by hand from memory, recovery time is however long that takes with people under pressure. This is the single highest-leverage investment in recoverability.

Secrets and configuration replicated. A standby environment that cannot authenticate to anything is not a standby.

Container images and artefacts available in both regions. A registry in the failed region means the standby cannot start.

Certificates valid in the standby.

Dependencies checked. Third-party services, payment processors and identity providers have their own regional architecture. A perfectly failed-over application that cannot reach its payment provider has not recovered.

Test it, on a schedule, deliberately

An untested failover plan is a document, not a capability. The failure modes are mundane and only surface when you try: a missing permission, an out-of-date image, a hardcoded region in a configuration file, a runbook referring to a system that was decommissioned.

Start with a tabletop walkthrough, which costs an hour and catches the obvious gaps. Progress to failing over a non-production environment. If your appetite and maturity allow, run a real failover in production during a planned window.

Whatever level you reach, do it on a recurring schedule, because environments drift and a plan verified two years ago is not verified.

The honest cost conversation

Resilience is bought, not engineered for free, and the price rises steeply with each level. It is legitimate for an organisation to decide that four hours of downtime once every few years is cheaper than permanent multi-region infrastructure.

What is not legitimate is making that decision implicitly, by never having the conversation, and then discovering the answer during an outage. Write the recovery numbers down, cost the levels, and let whoever owns the risk choose deliberately.

The same reasoning applies across providers. The catalogue of cloud accounts sets out where each one fits, if you are still deciding which to build on.

For the vendor’s own reference on the services involved here, see the OVHcloud documentation.

If you would rather test this than read about it, our OVHcloud accounts come in four configurations across a wide product range, and the cloud account catalogue lists every provider we carry.

Questions people ask