The Folly of Fixing It Live: Why Production Changes Have No Business in Office Hours
Somewhere, right now, someone is about to run a patch against a production system at 2pm on a Tuesday, while the business is open, the users are logged in, and the revenue is flowing. No Change Advisory Board. No rollback plan. Just a terminal, a deadline, and the quiet confidence that this one will be fine. It will not always be fine. And when it is not, the outage will not happen in a quiet maintenance window at 2am where nobody notices — it will happen in front of every customer, every stakeholder, and every director who ever doubted IT could be trusted to run itself.
ITIL Never Said "Do It During The Rush"
ITIL v3's Service Transition volume is explicit about this: every Request for Change (RFC) is assessed against the Seven Rs of Change Management before it is ever scheduled:
- Raised — who raised the change?
- Reason — what is the reason for it?
- Return — what return is expected from it?
- Risks — what are the risks involved?
- Resources — what resources are required?
- Responsible — who is responsible for building, testing, and implementing it?
- Relationship — what is the relationship between this change and other changes?
Nowhere in that assessment does "the users happened to be online at the time" appear as a mitigating factor. It appears as a risk to be minimised, not a convenience to be ignored.
A Request for Change exists to force the question nobody wants to ask under deadline pressure: what happens if this goes wrong, right now?
The Change Advisory Board Exists For A Reason
The CAB is not bureaucracy for its own sake. It is the institutional memory of every change that has ever gone wrong, applied in advance to the one you are about to make.
Change Advisory Board
A cross-functional review that asks the uncomfortable questions before the change ships — not after it has taken down the order system. Its value is entirely in the word advisory: it exists to be consulted before the risk is taken, not briefed after the incident review.
Change Types
ITIL classifies changes as Standard (pre-approved, low-risk, repeatable), Normal (requires assessment and authorisation), or Emergency (urgent, but still authorised — never silent). Patching production mid-shift under deadline pressure is almost never a Standard change. It is usually a Normal change being smuggled through as if it were one.
Change & Maintenance Windows
A defined window exists precisely so that risk and business impact are decoupled. Outside that window, a failed change is an incident. Inside it, a failed change is a controlled rollback with nobody watching. The window is not red tape — it is the blast shield.
| Change Type | Risk Profile | Approval Path | Appropriate Timing |
|---|---|---|---|
| Standard | Low, well-understood, previously proven | Pre-authorised by policy | Can run in hours — but only because the risk was already retired |
| Normal | Moderate to high, assessed case-by-case | CAB or delegated authority | Scheduled maintenance window, outside peak load |
| Emergency | High, but the cost of inaction exceeds the cost of action | Emergency CAB (ECAB) — fast, not absent | As soon as authorised — with a rollback plan already written, not improvised |
What Happens When You Patch At 2pm
A change made while the business is live does not fail quietly. It fails in front of an audience. The support queue fills before the rollback finishes. The incident bridge opens before the root cause is understood. And the sentence every engineer dreads — "can we just roll it back?" — is asked by someone who was never told a rollback plan did not exist.
The cost of a change is not paid at deployment. It is paid at the moment it fails — and office hours is the most expensive possible moment to pay it.
Signs Your Organisation Is Patching Live Instead of Managing Change
- Changes are pushed to production during business hours because "it should be quick"
- There is no written rollback plan — only an assumption that one can be improvised
- The CAB is treated as a rubber stamp to collect after the fact, not a review to pass before
- "Emergency change" has quietly become the default label for "change we did not plan properly"
- Nobody can say, before the change runs, what the blast radius is if it fails
- Every Normal change is assessed against the Seven Rs before it is scheduled
- Changes run in a defined window, outside peak load, with monitoring already in place
- A rollback plan exists in writing before the change starts, not after it fails
- Emergency changes still go through an Emergency CAB — fast is not the same as unauthorised
The ITIL 4 Angle: Value Streams Do Not Forgive Cowboys
ITIL 4 renamed Change Management to Change Enablement — a small wording change with a large intent behind it. The point is no longer just control for its own sake; it is enabling change to move through the value stream as fast as is safe, and not one moment faster. ITIL 4's four dimensions of service management — organisations and people, information and technology, partners and suppliers, and value streams and processes — all sit downstream of a single failed change made at the wrong hour. A patch applied at 2pm does not just risk the system. It risks the trust that lets the next change be approved quickly, because now every future request is read through the memory of this one.
Nobody has ever regretted waiting for the maintenance window. Plenty of people have regretted not waiting for it. The CAB, the change window, and the rollback plan are not obstacles between you and the fix — they are the only reason the fix gets to stay fixed.
Further Reading
This article draws on the ITIL v3 Service Transition change management process and the ITIL 4 Change Enablement practice, both stewarded by PeopleCert.