Every data center has one document that specifies how the facility behaves when things go wrong: the sequence of operations (SOO). It defines what happens in the first ten seconds after utility power is lost, which generator carries which bus, when the uninterruptible power supply (UPS) rides through on battery and under what conditions it hands the load to bypass, and in what order the plant comes back after an outage. It is, quite literally, the specification for facility behavior under stress.
Yet on many programs it is the least controlled document on the project. The engineer of record issues a version during design. The controls contractor marks it up in submittals. Someone revises it by email during construction. The programmer codes against one copy, the test team writes scripts against another, and by substantial completion nobody can say with confidence which revision the running code was actually built from. The one document everyone depends on is the one nobody formally verifies before code gets written against it.
Regulated life sciences solved this problem decades ago, because it had no choice. Under good manufacturing practice (GMP) rules, a pharmaceutical facility cannot release product until its specification chain has been qualified end to end. The discipline that enforces this, commissioning, qualification and validation (CQV), transfers to data centers cleanly once the paperwork is sized to the risk. Here is how.
How GMP treats a specification
GMP verification is organized around the V-model, and the idea is plainer than the name suggests. Down the left side of the V, requirements decompose: a user requirements specification (URS) states what the facility must do, the design translates those requirements into systems, and detailed specifications define equipment and code. Up the right side, verification climbs back level by level, and every requirement on the left is traced to a test on the right that proves it. A requirement without a test is not done. A test without a requirement is noise.
Before anything is built, design qualification (DQ) asks a blunt question: does the design, on paper, meet the requirements? Reviewers walk the specifications against the URS and record every gap. DQ is the cheapest test the project will ever run, because a failure found there costs a markup, while the same failure found in the field costs rework, schedule and sometimes demolition.
Between design and the qualification protocols sit two acceptance gates. Factory acceptance testing (FAT) proves equipment performs at the manufacturer before it ships. Site acceptance testing (SAT) proves it still performs after shipping, rigging and installation in its real home.
After equipment arrives, three distinct questions follow, each with its own protocol. Installation qualification (IQ) verifies the right equipment was installed correctly: model numbers, materials, calibrations, terminations. Operational qualification (OQ) verifies each system does what its specification says across its full operating range. Performance qualification (PQ) verifies the integrated process holds its requirements under real load, over time. A traceability matrix maps every requirement to the test that verified it, so an auditor can establish in an afternoon that nothing was skipped.
Then there is change control. When a GMP specification changes, the change triggers a documented impact review: what does this revision touch, which completed tests does it invalidate, what must be repeated before the change is closed. A revised specification is an event with consequences, never a quiet PDF swap on a file share.
The requirements chain a data center already has
Data centers already own the left side of the V; most projects simply decline to treat it that way. The owner's project requirements (OPR) state what the facility must achieve: availability targets, concurrent maintainability, redundancy. The basis of design (BOD) explains how the design intends to achieve it. The SOO sits at the bottom of that chain as the lowest-level specification, the one that both the controls code and the functional test scripts inherit from. That last point deserves a beat: if the SOO is ambiguous, the code and the tests can agree with each other perfectly and both be wrong against the OPR.
The commissioning (Cx) levels supply the right side of the V, with Level 4 functional performance testing (FPT) verifying individual systems and Level 5 integrated systems testing (IST) verifying the facility as a whole. Lined up against GMP, the mapping is nearly one to one, and the honest version includes a column for where it is not.
| GMP element | Question it answers | Data center equivalent | Where the fit is loose |
|---|---|---|---|
| URS | What must the facility do? | OPR, with the BOD one level down | Data center OPRs are often thinner than a URS and less formally maintained |
| DQ | Does the design meet requirements before it is built? | Design-phase Cx review against the OPR, including a structured SOO review | Unofficial on many programs, and skipped when Cx is hired late |
| FAT | Does the equipment perform at the factory? | Level 1 factory witness testing | Clean |
| SAT | Does it still perform on site? | Site receipt, Level 2 checks, Level 3 startup | No single equivalent; the content spreads across three steps |
| IQ | Is the right equipment installed correctly? | Level 2 installation verification | Level 3 startup bridges toward OQ territory |
| OQ | Does each system perform to its specification? | Level 4 FPT | Clean |
| PQ | Does the integrated facility hold requirements under load, over time? | Level 5 IST with endurance runs | Clean only when IST includes sustained running at design load |
| Traceability matrix | Was anything skipped? | SOO-to-test matrix spanning Levels 4 and 5 | Mandatory in GMP; owner-optional in data centers |
| Change control | What does a revision invalidate? | Cx plan change process with retest decisions | Regulatory obligation in GMP; a choice the owner writes into the Cx plan |
Where the mapping is clean, and where it is not
An honest crosswalk admits its loose joints. FAT to Level 1 is the cleanest pairing on the board: same activity, same location, same logic of catching deviations while the vendor still owns them. OQ to Level 4 is nearly as clean, since both are scripted proof that a system performs to its specification, alarms and interlocks included.
SAT is the rung that surprises people on both sides. Pharma treats it as one named event; data centers smear the same content across site receipt inspections, Level 2 installation checks and Level 3 startup. Neither approach is wrong, but a GMP-trained reviewer looking for "the SAT" in a Cx plan will not find it under that name, and a data center hand reading a pharma protocol may not realize SAT contains work three different Cx levels would claim.
PQ deserves the most attention, because it exposes the habit data centers should most want to borrow. PQ has an over-time element built into its definition: the facility must hold requirements under real conditions for a sustained, defined period. Level 5 IST maps to it only when the program includes endurance running at design load, and that is precisely the part schedule pressure trims first. A pull-the-plug scenario that passes once at 2 p.m. is not the same evidence as a facility that held design load through a full thermal cycle.
The differences matter as much as the mapping. GMP qualification operates under a quality unit and a regulator with inspection rights; data centers have the owner's acceptance signature and the market's judgment. In GMP, change control is an obligation. In a data center, the same discipline exists only if the owner writes it into the Cx plan, which is exactly what the rest of this article argues for. For owners running hybrid campuses, with GMP suites or cleanrooms alongside data halls, the crosswalk buys one verification language across both. It also supplies the interview question that works in either vocabulary: show me your traceability.
Validate the SOO as a document first
The highest-value step on that list is the one that costs the least: a structured design review of the SOO before a line of code is written against it. This is DQ applied to controls. Put the engineer of record, the controls programmer, the Cx team and the owner's operations staff in one room with three references open: the electrical one-line, the equipment submittals and the OPR. Then walk every state and every transition, line by line. Utility loss. Generator start failure, including what the sequence does while the second unit cranks. UPS on bypass during a source transfer. Loss of a cooling string at design load. The restart hierarchy after a full outage. Every maintenance mode the operators will actually use.
The review exists to surface ambiguity, and ambiguity hides in predictable places:
- What "fail" means for a sensor. Out of range, stale value, offline: are those the same failure, and does the sequence respond the same way to each?
- Timer values and deadbands the narrative never states. The programmer will pick defaults, and the defaults quietly become the design.
- Overlapping failures. A chiller trips in the middle of a utility transfer, or a generator fails to start while its neighbor is in maintenance mode. If the document is silent, the behavior is undefined, which means it is defined by accident.
- Restart order when power returns. Which loads come back first, on what interlocks and delays, and what prevents the plant from slamming back all at once.
Every one of those is a design decision. If the SOO does not make it, the programmer will, alone, at a workstation, without the one-line in view.
Write lines you can test
The second habit is structural. Write every SOO line as an atomic, testable statement with four parts: condition, action, expected response, time limit. For example: on loss of utility voltage below 85 percent for more than two seconds, the transfer controls in the paralleling switchgear shall command generator start; the generator shall accept load within ten seconds; failure to reach rated voltage in that window shall initiate the reserve unit and raise a critical alarm at the building management system (BMS). Every clause in that sentence is a pass or fail step waiting to be executed.
Compare that with what sequences too often say: the system shall restart in a controlled manner. Controlled by what, in what order, within how long? A test writer cannot score that line, and a programmer cannot implement it without inventing the missing half.
Trace every line to a test
The traceability matrix is a spreadsheet, not a philosophy, and it ports directly. Every SOO line maps to at least one Level 4 test step or one Level 5 scenario. Every alarm in the sequence maps to a verification that it annunciates where it should, at the right priority. Then read the matrix backward. Anything unmapped is one of two things: untested behavior, which is a hole in the Cx plan, or untestable text, which is a hole in the specification. Both are far cheaper to find in a conference room than at three in the morning during IST.
Change control until the last test passes
Sequences change during construction. Equipment substitutions, coordination study results, field conditions and operator preferences all push revisions, and none of that is the problem. The problem is invisible change. Version the SOO the way software is versioned: revision number, date, author, change log, with the engineer of record approving every revision and the Cx team dispositioning retest. Review the impact of every change against testing already completed, and let the change decide what gets retested. A revised generator start timer after Level 4 reopens the Level 4 scripts that exercised it. A new restart hierarchy reopens the IST scenario that proved the old one.
What the rigor buys you
The return shows up exactly where programs usually hurt. Fewer surprises at IST, because the failure scenarios were walked on paper months earlier and the arguments happened in a review room instead of on the roof. A faster IST window, because scripts trace to sequence lines nobody disputes and retest scope is a lookup, not a negotiation. An acceptance package the operations team can trust: not a binder of signatures, but a matrix demonstrating that every behavior the owner paid for was tested and passed. And a defensible record, because when a breaker misoperates two years from now and someone asks whether that sequence was ever really verified, the answer is a document, not a recollection.
The economics justify the effort many times over. A serious outage at a hyperscale site can cost more per hour than the entire commissioning contract, which makes document-level rigor one of the cheapest forms of insurance an owner can buy. Skapa MC carried this discipline into mission critical from GMP life sciences, where our CQV specialists qualified facilities that could not ship product without it, and we have since applied it across 284 MW of commissioned hyperscale capacity, with another 134 MW in progress today. None of it requires a regulator. It requires an owner who writes it into the Cx plan: validate the sequence before code is written against it, trace every line to a test, and let change control, not optimism, decide what gets retested.