Risks and progress

Incidents

Everything that can go wrong: grid outages, power sags, feed and carrier cuts, smoke and fire, floods, earthquakes, burglaries, DDoS, frozen servers.

Real datacenters have bad days, and so does yours. Incidents happen on their own, around the clock, a little more often as your company grows (see Eras). New companies get a gentler world while they learn. Each one is announced in the operations log and, if it matters, with a notification.

Power

  • Grid outage: the building loses mains power. If its generators start, nothing happens; otherwise every cabinet in it goes dark. Only the building's tier helps.
  • Power sag: a short dip that reboots servers, and can burn a power supply. Only Unrated buildings let it through; a rack UPS or a B-feed protects you there.
  • Power feed cut: your cabinet's power line drops for hours. A B-feed keeps servers with two power supplies running.
  • Breaker tripped: you drew too much; the cabinet goes dark for a while (see Power and cooling).

Network

  • Uplink cut: the carrier's line is cut and the cabinet is offline. A second carrier keeps it up.
  • DDoS attack: a flood of junk traffic fills your line. A DDoS filter or an IPS firewall stops it (see Networking).

Fire

A smoke alarm gives you a short time to react. You get an alarm with a countdown and an Emergency power-down button:

  • Power down: the cabinet is dark for a couple of hours while the building checks it. Nothing burns.
  • Do nothing: it may turn into a fire. How much hardware burns depends on the building's fire protection, from almost everything with hand extinguishers to nothing with gas suppression.

A NOC operator on shift answers alarms for you, even at night. Hot cabinets raise smoke alarms more often. Tier 3 buildings deal with smoke on their own.

Nature

  • Flood: water reaches the lowest units of cabinets in buildings without a raised floor, destroying parts in servers mounted there. Some cities flood more than others.
  • Earthquake: racks are bolted down, so nothing breaks, but it knocks out one power feed for hours, except in a braced Tier 3 building. Servers on a B-feed with two power supplies keep running.

People

  • Burglary: a burglar takes a few parts from one server. They are marked STOLEN; clear the record (allowed even while the server runs, since the part is gone) and fit replacements. Processors and memory still need the server powered down to go in. A cabinet lock stops most burglars.
  • Staff mistake: facility staff unplug one of your servers for a few hours. Happens most in Unrated buildings, never in Tier 3.
  • Frozen server: an operating system hangs. A remote console reboots it in minutes; without one, remote hands get to it after a while.

Wear

Parts also simply wear out and fail. See Maintenance. A dead part can take a server down, or leave its customers without enough memory: replace it within 2 hours or the customers who no longer fit leave (see Server parts).

What protects you

Incident What helps
Grid outage A higher-tier building
Power sag Tier 1 or higher, a rack UPS, or a B-feed
Feed cut, earthquake B-feed and two power supplies per server
Carrier cut A second carrier (and a BGP router)
DDoS DDoS filter, IPS firewall
Fire NOC operator, environment sensor, good fire protection
Flood Raised floor (Tier 2+), servers mounted higher
Burglary Cabinet lock, better building security
Lost data RAID and a backup server
Lost hardware Insurance

Common mistakes

  • Everything in one cheap building in a risky city.
  • No NOC operator and no sensor: smoke alarms at night are left to chance.
  • A B-feed with single-supply servers.

Want the numbers? The full rules: Surviving incidents

Start a hosting company with $50,000

Free to play in your browser. One shared world, running in real time.