Incidents in ColoTycoon happen on the real clock, also while you are logged out. Each one takes servers offline, destroys parts or both, and every customer on an affected server decides on the spot whether to leave. Most of the damage is decided before the incident starts, by the building you chose and what you put in your cabinet. This guide covers the preparation first, then what each incident does and how to respond.

Before anything happens

Work through these steps in order. Insurance has a waiting period, so it must be in place well before you need it.

Step 1: Choose the building with the city in mind

The facility tier decides whether grid outages, power sags, fires, floods, quakes and staff errors reach you at all. Check the city's flood and earthquake risk on the provider's card before renting. The full comparison is in choosing a tier.

Step 2: Give servers two power supplies and add a B-feed

On the Racks page, the Providers section lets you Add B-feed for 30% more rent. When one power feed drops, servers with two power supplies stay up on the other. Two supplies count as redundant only if either one can carry the server's load alone; the Nodes page shows the server as redundant or single. A B-feed also rides through power sags.

Step 3: Add a second carrier

Add second carrier costs 50% more rent and adds an uplink port. When the carrier cuts the circuit, the second carrier keeps your cabinet online.

Step 4: Fit a rack UPS where the building has none

Unrated buildings have no building UPS. Mount a rack UPS rated above the cabinet's total draw: 1 kW (1U, $650), 2.7 kW (2U, $1,600) or 5.4 kW (4U, $3,800). The Protection strip under the cabinet's gauges shows whether the UPS covers the load. A UPS that is too small does nothing.

Step 5: Lock the cabinet

A cabinet lock costs $25 a month, billed with the rent, and stops 90% of burglaries.

Step 6: Take out hardware insurance

Open Finance and pick a plan in the Hardware insurance panel. Premiums are a monthly share of the book value of your racked and stored hardware, with a minimum of $10 a month. A billing month passes every 24 real hours, so the full monthly premium is billed once a day.

Plan Premium Deductible Covers
Standard 0.15% of insured value a month $1,000 fire, flood, theft
Premium 0.25% of insured value a month $250 the same, plus power surges

Claims are paid only after a 72-hour waiting period from the day you sign up, so take the policy before you need it. A claim pays the book value of what was destroyed minus the deductible; lost revenue and customers are not covered. See insurance and loans.

Step 7: Use RAID and a storage server for backups

On the Nodes page, set a Storage layout that tolerates a disk failure: RAID 1 (2 disks or more, an even number), RAID 5 (3 or more, tolerates one) or RAID 10 (4 or more, an even number). No RAID and RAID 0 tolerate nothing.

Then add a backup target. The quickest is a rack NAS ($650): a closed 1U box with 500 GB of backup space that only needs to be mounted and cabled to a switch. For more space, build a storage server: a storage chassis full of disks, racked and running the Storage Server OS ($25 a month). On each server, click Set backup server and pick it. The backup must hold the data in use on every server attached to it (for a dedicated server the game assumes half its disk is in use); servers that do not fit stay unprotected. If an array dies with no redundancy left, a backed-up server keeps its customers' data. Without a backup, the data is wiped, your reputation drops by 5 points and each customer on it leaves with a 50% chance.

Step 8: Hire staff to cover the hours you are away

Open Staff. Two roles exist:

  • A technician replaces failed and stolen parts with a spare of the same model from your inventory, changing hot-swap parts live and shutting the server down briefly for the rest. Each job takes 75 minutes minus 7 per skill level.
  • A NOC operator powers down a cabinet when a smoke alarm goes off. Each job takes 20 minutes minus 3 per skill level, so even a skill 1 operator answers inside the 45-minute window.

Everyone works one 8-hour shift (00-08, 08-16 or 16-24 UTC), so round-the-clock cover takes three people per role, up to 6 per role. A technician's salary is $2,500 a month plus $400 per skill level, an operator's $2,000 plus $300, each varied by up to 10% per candidate. Payroll runs on the billing clock: it is paid once a day, and each day's payroll is a full month's salaries. If you cannot pay it, all your staff walk out. Under Automation rules you can let technicians buy a new spare from the vendor when none is in stock, up to a price limit ($2,500 by default). See staff.

Fires and the response window

A smoke alarm fills the alarm band at the top of every page with a countdown. You have 45 minutes to click Emergency power-down. If you do, the cabinet goes dark for about 2 hours while the facility finds the smoke, and nothing burns. If you do not, the smoke turns into a fire 60% of the time. A fire destroys a share of the parts in your cabinet (35% with pre-action sprinklers, 70% with wet sprinklers, 90% with hand extinguishers only) and keeps the power off for 6, 12 or 18 hours. Tier 3 buildings catch smoke early and handle it without involving you.

Smoke is 3 times as likely in a warm cabinet and 8 times as likely in an overheating one, so keeping heat under cooling is fire prevention.

Outages

  • Grid outage: about once every 35 days per building, 1 to 4 hours. Only a working generator prevents it.
  • Power sag: about once every 10 days per building. Absorbed by a building UPS (Tier 1 and above), a rack UPS that covers the load, or a B-feed. Otherwise every server reboots, about an hour down, and each has an 8% chance of losing a power supply.
  • Power feed down: about once every 70 days per cabinet, 1 to 3 hours. Servers with two power supplies stay up on a B-feed.
  • Uplink cut: about once every 40 days per cabinet, 1 to 3 hours. A second carrier carries the traffic.
  • Staff error: facility staff knock out the power cord of one server for 1 to 3 hours. Never in Tier 3.

Floods and earthquakes

Floods and quakes depend on the city. A flood without a raised floor destroys 60% of the parts in the lowest 6 units of your cabinet and keeps it dark for 4 to 12 hours, so keep empty space or cheap gear at the bottom in flood cities. An earthquake outside a braced Tier 3 building trips one power feed for 2 to 6 hours: servers on a B-feed with two power supplies, behind switches with two supplies, stay up; everything else goes dark. Nothing is destroyed.

Burglaries

Burglars take 1 to 3 parts from one server. The parts are marked STOLEN in the server and the slot is physically empty: remove the record (allowed even while the server runs) and fit a replacement, or let a technician do it. A replacement processor, memory module, M.2 drive or card still needs the server powered down.

DDoS attacks

A cabinet with paying customers is attacked about once every 28 days, at 2 to 60 Gbps for 1 to 3 hours. Unless a firewall with intrusion prevention sits on the uplink and its throughput exceeds the attack, the uplink is saturated and every server in the cabinet goes offline. The Branch Firewall filters up to 5 Gbps ($2,500), the Edge Firewall 10G up to 20 Gbps ($9,000) and the Datacenter Firewall 100G up to 120 Gbps ($45,000). The Small Office Firewall has no intrusion prevention. Cable the firewall to the uplink and your switch to the firewall; the network page explains the chain.

Part failures

Parts wear on the real clock, faster under load and much faster in a hot cabinet. Below 40% health their failure odds climb steeply; the Maintenance page lists every part that has failed or is below 40%, with the time left.

  • Drive bays, power supplies and blade bays are hot-swap: replace them while the server runs.
  • On a server with customers, a power supply can come out only if the others carry the load, and a disk only if its array has redundancy to spare.
  • Processors, memory, M.2 drives and PCIe cards are cold-swap: the game will not fit or pull them while the server runs. Schedule a maintenance window at least 24 hours ahead (15, 30, 60 or 120 minutes long); the time does not count as downtime. A forced shutdown needs no window but counts as downtime and powers back on after 60 minutes.

Keep spares in inventory so a technician can swap a failed part without you.

After an incident

The Overview operations log records every incident and how many customers left after it. Losses and payouts appear on the Finance page under Hardware insurance. Reputation falls by 0.3 points for every hour a customer's server is down, so bringing servers back quickly matters as much as preventing the incident.