Self-assessment and resilience: looking beneath the ‘outcome’

The NCSC Cyber Assessment Framework provides an extremely useful basis for assessing cyber resilience. Its strength is that it is outcomes-based: it defines what an organisation needs to achieve without prescribing a single way of achieving it.

That flexibility is important. Organisations differ in their technologies, operating environments, risk profiles and management arrangements. However, it also creates a challenge. Because the CAF is sector agnostic, an assessment cannot stop at deciding whether an outcome appears to have been achieved. It must examine the management functions that deliver and sustain that outcome in the organisation’s particular operating context.

Those functions include governance, policies, processes and procedures, risk assessment, technical and procedural controls, competence, monitoring, incident response and continual improvement. For operational technology, they must also reflect what the systems and controls are protecting: physical processes, hazardous inventories, critical equipment and, in some cases, safety instrumented functions.

The harder question is therefore not simply, is this outcome achieved? It’s do the underlying arrangements provide sufficient evidence that the outcome is being—and will continue to be—achieved?

Outcomes depend on interconnected functions

Answering that question requires more than a checklist because management functions do not operate independently.

Process safety provides a useful analogy. Effective emergency response depends on competent people and reliable assets. Those depend on training, inspection, testing and mechanical integrity. Changes affecting any of them must be identified and controlled through management of change, which must itself consider the consequences for emergency response.

These functions loop into and depend on one another. An emergency response procedure may exist and appear complete, while changes to plant, personnel or equipment have quietly undermined the organisation’s ability to implement it.

OT cyber resilience has the same shape. A control system asset may support an alarm, an interlock, a safety instrumented function or an emergency shutdown. Its resilience can depend on asset management, system architecture, access control, supplier arrangements, vulnerability management, monitoring, competence, incident response and recovery planning. A weakness in any one of these functions may affect several others.

An outcome can therefore appear to have been achieved when considered in isolation, even though the arrangements supporting it are incomplete, poorly connected or no longer effective.

This is why self-assessment should not revolve around a flat list of questions that are answered once and marked complete. It needs a methodology capable of examining each outcome in sufficient detail, identifying the management functions that support it, understanding the relationships between them and recording the evidence on which the assessment is based.

What the inspection evidence shows

HSE’s OT cyber trial inspection programme illustrates the difference between assumed assurance and evidence-based assurance.

The programme covered volunteer COMAH operators from chemical manufacturing, refining, fuel pipelines, gas terminals and industrial gases. Despite being a self-selecting group that had already taken an interest in OT cyber security, none had fully achieved all the objectives expected for good cyber hygiene. HSE concluded that compliance across the wider sector was likely to be lower still.

Importantly, operators that made progress did so by engaging with the guidance and inspection process and working through what the requirements meant in practice. Greater awareness of cyber risk was not enough on its own. Improvement came from applying structure: examining the arrangements behind the required outcomes, identifying gaps and acting on the findings.

The underlying problem is often not that an organisation has consciously decided the risk is acceptable. More commonly, responsibility sits somewhere between IT, engineering, operations and process safety without being fully owned by any of them. In that space, assumptions can easily take the place of assurance.

“We have never had an incident” is not evidence that the controls are effective. It may simply mean that the organisation has not detected one.

Visibility matters

Triton demonstrated what an absence of visibility can mean in practice. The malware targeted the safety instrumented system at a petrochemical facility and was discovered only after an error in its own code caused the safety controllers to enter a protective state.

Triton remains important not simply because it was a cyberattack, but because it crossed directly into process safety. A safety instrumented system is process safety equipment. Malware capable of interfering with it creates a process safety threat that has arrived through the network.

Attacks that successfully reach and manipulate a safety system remain rare, even though disruptive activity affecting industrial organisations continues to grow. That rarity is not reassuring. It means there are relatively few events from which organisations can learn, while weaknesses in architecture, access control, monitoring or supplier management may remain undetected until something else goes wrong.

This reinforces the need to assess not only whether controls exist, but whether the organisation can demonstrate that they are appropriately designed, implemented, maintained and monitored.

From checklist to evidence-based assessment

A credible self-assessment should therefore connect four things:

  • the cyber resilience outcome the organisation needs to achieve;
  • the management and technical functions that support it;
  • the relationships and dependencies between those functions; and
  • the evidence demonstrating that the arrangements work in practice.

That evidence might include policies and procedures, defined responsibilities, risk assessments, network architecture, asset records, access reviews, maintenance and testing records, competence records, supplier controls, monitoring outputs, exercise findings and completed improvement actions. No single item proves that an outcome has been achieved. Assurance comes from considering the evidence collectively and testing whether the different parts of the system support one another.

Because these relationships are complex, the assessment methodology matters. A relational structure capable of connecting outcomes, requirements, controls, evidence, findings and improvement actions makes that complexity manageable in a way that a flat checklist or spreadsheet cannot. It also allows an organisation to see how a weakness in one area affects other parts of the management system.

The assessment must also bring the right disciplines together. IT and cyber expertise are essential, but so are operational, engineering and process safety knowledge. Assessors need to understand not only the technical control, but also the process hazard or operational consequence it is intended to prevent or mitigate.

Self-assessment as organisational understanding

Done properly, self-assessment can provide more value than a one-off audit or inspection. It allows the organisation to look beneath the outcome, understand how its arrangements operate, identify where functions are disconnected and track whether improvement actions produce a stronger result over successive assessment cycles.

That is very different from completing a checklist or recording a single pass-or-fail judgement. It creates a defensible body of evidence showing what has been assessed, why conclusions were reached, where weaknesses remain and how resilience is improving over time.

The CAF provides the outcomes. Sector-specific guidance helps interpret what those outcomes mean for OT environments. A structured, evidence-based methodology connects them to the management functions, technical controls and process safety dependencies that determine whether the outcomes can genuinely be achieved.

That is what turns self-assessment from a compliance exercise into organisational understanding—and it is the discipline around which OpenPSM is built.

Further reading