5 days of ICS & SCADA security in Amsterdam

This week I have been present at SANS ICS Security Amsterdam, taking one of their courses on security for industrial control systems. This has been a fantastic opportunity to learn new things, reinforce known concepts at a deeper level, and to network and meet with a large range of people with interests in this field. I’ve been surprised to see people from law enforcement, national security, industry representatives, consultants and vendors coming together at one event. Information security has been a lot in the news lately, and is seen as a big part of the risk picture in almost every region and every industry.

Any security training and seminar needs its own T-shirt. And soft drinks.

Two things in particular have been very interesting to see when it comes to the stuff presented by the SANS course instructors:

  1. Security researchers continue to find basic vulnerabilities in new product lines from major vendors (the big companies we’ve all heard about, I’m not going to shame anyone)
  2. A lot of control systems are still facing the internet, are directly accessible with no or very weak security, and attacks are prevalent as found in honeypot research experiments

Basically, this confirms the notion that “the situation is bad and we need to do something about it”. People said this after Stuxnet, and they are still saying it. My impression from working with various clients is that industry is aware of the risks that exist “out there” but they are in varying degree doing something to control that risk. Too many still believe that “we will never be compromised as long as we have a firewall”. Relating to this, one might ask, what are the “basic vulnerabilities” and how do we work around that?

Many control system components today run on commodity OS’s, or are connected to servers running MS Windows or Linux, e.g. used to display HMI’s. These HMI’s are in many modern systems developed as web apps (running on local servers) for portability, ease of access, etc. This means that many of the vulnerabilities found in regular IT and on the web also apply to control systems. However, these risks are worse in the control system world because these systems need to run all the time and can therefore often not be patched, and should someone break in, they could cause real physical damage (think crashing of cars, blowing up an oil rig or destroying a melting furnace). Some of the top vulnerabilities we are exposed to are the following: buffer overflows (yes, still, lots of stuff running on old systems), SQL injection vulnerabilities and cross-site scripting vulnerabilities (web interfaces…). So, if we cannot patch, what can we do about this?

First of all, perform a risk and vulnerability assessment, taking both possible scenarios and credibility of scenarios into account. Make sure to establish a good baseline security policy and use this for managing these issues – there is lots of guidance available, and often sector specific. If you cannot patch, focus on what you can do; ensure everyone involved in purchasing, maintaining, producing and using control systems are aware of the risks and what types of behaviors are good, and what types are bad. This means that security awareness must be built into the organizational culture.

On the technical side, maybe especially with lots of legacy systems running, make sure the network architecture is reasonable and safe – avoid having critical assets directly facing the internet (do a Shodan search and you will find that lots of asset owners are not following good practice here). The architecture must weight risks and business needs – a full lockdown may be the safest way to go, but it may also stop core business functions from working.

Further, should a breach occur, make sure you have the organizational and technical capabilities to deal with that. Plan and train on incident response – and remember you are not alone. Get help from vendors both in managing the assets during normal operations, and during a crisis situation. Including incident response in service agreements may thus be a good idea.

This was a quick summary of topics we’ve looked at during training, and discussed over beers in Amsterdam. The training by SANS has been excellent, and I’m looking forward to bringing reinforced and new insights back to the office on Monday.

What does the IEC 61508 requirement to have a safetey management system mean for vendors?

All companies involved in the safety lifecycle are required to have a safety management system, according to IEC 61508. What the safety management process entails for a specific project is relatively clear from the standard, and is typicaly described in an overall functional safety management plan. It is, however, much less clear from the standard what is expected for a vendor producing a component that is used in a SIS, but that is a generic product rather than a specifically designed system for one particular situation.

For vendors, the safety management system should be extensive enough to support fulfillment of all four aspects of the SIL requirement the component is targeting:

  • Quantitative requirements (PFD/PFH)
  • Semi-quantitative and architectural requirements (HWFT, SFF, etc.)
  • Software requirements
  • Qualitative requirements (quality system, avoidance of systematic failures)
A great safety management system is tailored to maintain the safety integrity level capability of the product from all four perspectives. Maintaining this integrity requires a high-reliability organization, as well as knowledgable individuals.
A great safety management system is tailored to maintain the safety integrity level capability of the product from all four perspectives. Maintaining this integrity requires a high-reliability organization, as well as knowledgable individuals.

Quite often, system integrators and system owners experience challenges working with vendors. We’ve discussed this in previous posts, e.g. follow-up of vendors. Based on experience from several sides of the table, the following parts of a safety management system are found to be essential:

  • A good system for receiving feedback and using experience data to improve the product
  • Clear role descriptions, competence requirements and a training system to make sure all employees are qualified for their roles
  • A good change management system, ensuring impact of changes is looked at from several angles
  • A quality system that ensures continuous imrovement can occur, and that such processes are documented
  • A documentation system that ensures the capabilities of the product can be documented in a trusted way, taking all changes into account in a transparent manner

A vendor that has such systems in place will have a much greater chance of delivering top quality products – than a vendor that only focuses on the technology itself. Ultra-reliable products require great organizations to stay ultra-reliable throughout the entire lifecycle.

Profiling of hackers presented at ESREL 2015

Yesterday my LR colleague Anders presented our work on aggressor profiling for use in security analysis at the European risk and reliability conference in Zürich. The approach attracted a lot of interest, also from people not working with security. One of the big challenges in integrating security assessments into existing risk management framework is how to work with the notion of probability or likelihood when considering infosec risks. Basically – we don’t know how to quality the probability of a given scenario in a reasonable manner – so how can we then risk assess it and treat it in a rational manner?

The approach presented looks exactly at this. A typical risk management process would involve risk identification, analysis and evaluation of consequences and likelihoods, planning of mitigation and follow-up/stakeholder involvement. We have found working with clients that people find identifying potential consequences of different scenarios much easier than identifying the credibility of scenarios. The approach to assessing credibilities is centered around two actors:

  1. Who is the victim of the crime?
  2. Who is the aggressor?

Given a certain victim with its financial standing, relationships to other organizations, geopolitical factors, etc., we can form an opinion about who would have any motivation to try and attack the asset. Possible categories of such attackers may be

  • Script kiddies
  • Hacktivists
  • Other corporations
  • Nation states
  • Terrorists
  • Rogue internals

Each of these stereotypes would have different traits and triggers shaping the credibility of an attack from them. This is related to motivation or intent, their resources and stamina, their skill sets and the cost-benefit ratio as seen from the bad guy perspective. Giving scores to these different traits and triggers can help establish the opinion of how credible a threat is.

An interesting effect in security is that the likelihood of a threat scenario is not necessarily decoupled from the consequence of the scenario; the motivation of the perpetrator may be reinforced by the potential gains of great damage. This should be kept in mind during considerations of intent and cost-benefit.

Forming structured opinions about this, allows us to sort threat scenarios not only according to consequences, but also according to credibility. That fits into standard risk management framworks. Somewhat simplified we can make a matrix to sort the different threat scenarios into “acceptable”, “should be looked at” and “unacceptable”.

Managing information security should be a natural part of your risk management system

Anyone not hiding in a hole in the ground the last five years must have noticed how much media is writing about cyber threats. The media picture aligns well with the impression I get when talking to clients; business risks for production centered firms tend to no longer be dominated only by random accidents – but also by potential cyber threats. This means that such threats need to be managed.

Most companies already have a risk management system in place, typically along the lines of ISO 30001. A risk management process must consist of a cycle of activities;

  • Identify risks
  • Evaluate risks up against acceptance criteria for severity and probability
  • Plan mitigating actions
  • Involve internal and external stakeholders along the way

Exactly the same process can be used to deal with information security risks. A good methodology for managing such security risks should take into account who the threat actors are in the context of the business. Why do they want to attack? Do they have a lot of resources and know-how? What is the cost-benefit ratio for the bad guys? Understanding these human factors in the equation at play will arm you with a reasonable way to rank the credibility of various attack scenarios. This helps put cyber risks into the typical risk management scenarios; resources should be spent where they bring the largest risk reduction. Typically, highly credible attack scenarios with terrible consequences should be dealt with first, less likely or less severe scenarios second, and scenarios that seem extremely unlikely or have no or little impact on our business are perhaps OK. This sort of risk ranking is well-known to risk management professionals. A few differences on assessing credibility of attack scenarios from more random events are the following;

  • A bad guy may be motivated by worse consequences; the probability of an attack is thus not decoupled from the consequence
  • It is hard to use “probabilities” or “frequencies” in a meaningful sense – a qualitative approach may be just as useful (sometimes true also for random risks!)

Mitigations should thus be planned according to risk reduction needs and effects of the mitigation approaches. This is exactly the same as we do in other risk management settings. Also, communication with stakeholders is equally necessary in this context, if not even more. Risk owners, equipment suppliers, users, other supply chain partners – they can all be affected. And in our connected world, and increasingly so as we move to smarter production systems, cross-infection may be possible across various domains and interfaces. The tools of communication well-known to risk managers are therefore equally important when managing risks to production critical information systems.

Managing infosec requires a keen eye for the context of the business, and adaptability to new realities is a key success factor

The bottom line is: don’t outsource responsibility for risk management to your IT services provider – integrate cyber risk management into your existing risk management process. That is the only way to be in control of your own environment.

Would you operate a hydraulic machine with no cover or emergency stop?

Sounds like a crazy idea, right? Research, however, has shown that about half of professional machine operators do not think safety functions are necessary. You know, things like panel limit switches, torque limiters and emergency stop buttons. Who would need that, right? This number comes from a report issued in Germany about 10 years ago, but I am not very optimistic in improvements in these numbers since then. The report can be found here: http://www.dguv.de/ifa/Publikationen/Reports-Download/BGIA-Reports-2005-bis-2006/Report-Manipulation-von-Schutzeinrichtungen/index.jsp (in German).

Machine safety is important for both users and bystanders. Manipulations of safety functions are common – and the risk increase is typically unknown to users and others. How can we avoid putting our people at risk due to degraded safety of our machinery?

Researchers have found that the safety functions of machines are frequently manipulated. This is typically done because workers perceive the manipulation as necessary to perform work, or to improve productivity. Everyone from machine builders, to purchasers to operrators should take this into account, to avoid accidents from happening. Consider for example a limit switch. A machine built to conform to the machinery directive (with CE marking) has to satisfy safety standards. Perhaps has a SIL 2 requirement been assigned to the limit switch because operation without it is deemed dangerous and a 100-fold risk reduction is necessary for operation to be acceptable. This means, if the limit switch is put out of function, the risk of operation is 100 times higher than the designer has intended!

What can we do about this? We need to design machine such that safety functions become part of the work flow – not an obstacle to it. If workers have nothing to gain in their own perception from manipulating the machine, they are not likely to do it either. This boils down to something we are aware of, but are not good enough at taking into account in design processes; usability testing is essential not only to make sure operators are happy with the ergonomics – it is also essential for the safety of the people using the machine!

Why functional safety audits are useful

Functional safety work usually involves a lot of people, and multiple organizations. One key success factor for design and operation of safety instrumented systems is the competence of the people involved in the safety lifecycle. In practice, when activities have been omitted, or the quality of the work is not acceptable, this is discovered in the functional safety assessment towards the end of the project, or worse, it is not discovered at all. The result of too low quality is lower integrity of the SIS as a barrier, and thus higher risk to people, assets and environment – without the asset owner being aware of this! Obviously a bad situation.

Knowing how to do your job is important in all phases of the safety lifecycle. Functional safety audits can be an important tool for verification – and for motivating organization’s to maintain good competence mangement systems for all relevant roles.

Competence management is a key part of functional safety management. In spite of this, many companies have less than desirable track records in this field. This may be due to ignorance, or maybe because some organizations view a «SIL» as a marketing designation rather than a real risk reduction measure. Either way – such a situation is unacceptable. One key tool for ensuring everybody involved understands what their responsibilities are, and makes an effort to learn what they need to know to actually secure the necessary system level integrity, is the use of functional safety audits. An auditing program should be available in all functional safety projects, with at least the following aspects:

  • A procedure for functional safety audits should exist
  • An internal auditing program should exist within each company involved in the safety lifecycle
  • Vendor auditing should be used to make sure suppliers are complying with functional safety requirements
  • All auditing programs should include aspects related to document control, management of change and competence management

Constructive auditing can be an invaluable part of building a positive organizational culture – where quality becomes as important to every function involved in the value chain – from the sales rep to the R&D engineer.

One day statements like “please take the chapter on competence out of the management plan, we don’t want any difficult questions about systems we do not have” may seem like an impossible absurdity.

Safety versus convenience 

Risk based asset management frameworks force us to be systematic in our approach. Multiple layers of defense are commonly applied to mitigate risks down to what we see as an acceptable level. In many cases it will feel like each layer of defense is a layer of inconvenience. 

 

Do we maximize the number of spikes (or layers of protection) to feel safe?

The ALARP principle is often used to evaluate if a certain defense layer is worth the investment. This type of analysis tends to be CAPEX focused. Very cumbersome operations tend to make people invent bypasses that are more convenient. In addition to investment cost, maybe we should also include the effect of the mitigation solution on convenience and how humans react to it, in addition to cost? If people bypass the intended operating procedure – the result of the new risk mitigation investment could be an increase in the overall risk.  

Is technical competence king in detecting phishing?

Human factors researchers have taken interest in cyber security. This is good, because we need to think about most attacks in terms of both technology and psychology on both sides of the fence. Phishing emails is the most common initial attack strategy used in targeted attacks. It is therefore important to make your people able to avoid such deception. 

 

Understanding the difference between gold and trash is the main way to avoid phishing
 
A recent paper in the August issue of “Human Factors” by Proctor and Chen discusses decision making in detection of phishing. A key factor found by researchers is that a mismatch between cues in a phishing email and the expectations the recipients have is crucial to detecting a phishing attempt. Such cues are typically technology related such as strange URL’s, errors in corporate identity, slight misuse of terminology. It may this be questioned if awareness training by itself is an effective mitigation element – people need to know their domains well too, as well as what to expect of URL’s and technology solutions from emails and web sites. 

Are you aware of the effect of work life balance issues on the quality of your team’s work?

Make sure your people do not feel like a hot kettle with nowhere to let the steam out – that can lead to broken designs – and if your line of work is designing safety critical systems, broken designs usually means a greater chance of loss of life, polluting the environment and large financial losses.

Having a way to control the internal steam pressure of your team members may be utopia - but you should still look for ways to avoid disasters, together with your people.
Having a way to control the internal steam pressure of your team members may be utopia – but you should still look for ways to avoid disasters, together with your people.

We all know that the quality of our work varies – with a large number of factors. If we are overworked or really worried about something in our personal lives – quality of our work will most likely suffer. If you are responsible for the functional safety in a large project, human error can be disastrous, not only for the project, but for the people working in the plant when it has become operational. Whether it is yourself, or an entire team that you are responsible for, you need to be aware of key performance shaping factors. These factors are described in detail in human reliability analysis, such as developed by Idaho National Labs for the nuclear industry: http://www.nrc.gov/reading-rm/doc-collections/nuregs/contract/cr6883/cr6883.pdf. These techniques can lend some terminology and thinking that is useful in the project itself, to help manage the risk of significant human errors in the project phase. Remember – misunderstanding the risk factors and barrier elements themselves may lead to insufficient barriers against major accident hazards in a real plant! The factors in the SPAR-H methodology described in the linked document are:

  • Available time
  • Stress/stressors
  • Complexity
  • Experience/training
  • Procedures
  • Ergonomics/HMI
  • Fitness for duty.
  • Work processes

These factors have been defined for typical process operators’ actions in a nuclear power plant but they are also relevant for other types of tasks. Functional safety work typically has a high degree of complexity. The experience and training of people involved in the safety lifecycle tend to vary a lot, and procedures and work processes are not always clear to everyone involved. All of this falls under “management of functional safety” and project managers should think about what creates great quality when planning and managing the project. In many projects, time is quite limited, and the term “schedule impact” is a rather frightening concept to many project managers. This can lead to tasks being perceived as less important simply because the schedule is prioritized over quality. For safety critical tasks, this should not be allowed to happen.

Some factors from your project members’ personal lives may have severe impact on performance. People working on the project team can be stressed or not “fit for duty” due to a number of challenges that are not only work related. How can we deal with this? Project managers need to know their teams beyond their tasks and work backgrounds. You need to create an environment of trust, such that you have a greater chance of catching such performance limiting factors originating from outside the organization. For many people these factors may not be something that is seen as “bad” such as divorce, alcohol abuse or depressions, it may simply be challenges in making daily life work. People tend to want balance in life – with room for work, family, friends, hobbies, etc., etc. Working in a high-stakes project may itself be a threat to a balanced life. By knowing your people you can help them find the necessary balance that will also improve their performance at work. Flexible work-hours, part-time telecommuting and close follow-up with real feedback to every member on your team can help.

We consider human factors and the effect of the work environment as well as external performance shaping factors for operators. We should also strive for people to perform at their best when their work is to design the very systems used by the operators after commissioning.

Stages of process safety understanding

Defining process safety should be quite straightforward. However, what people mean with this term can vary quite a lot, and what to include in the term depends a lot on the understanding people have of the anatomy of severe accidents. Personally, I have met the following different understandings of the topic:

  • Process safety is what is governed by API 521 (basically steel strength and dimensioning of pressure relief valves)
  • Process safety is the technical measures taken to stop an accident from occurring
  • Process safety is the sum of organizational and technical systems involved in mitigating risk of major accidents

The first statement is obviously too narrow – especially as we know that more than half of accidents are down to human factors! Definition number 2 is a traditional view, and slightly more mature as it includes both the safety instrumented system and alarm management (to a certain extent). The last definition is maybe the most “modern”, and includes organizational culture, safety leadership as well as the technologies included in the first and second definitions.

How people understand the term “process safety” tends to mature over time – from a strictly technical view to a more holistic view including both individual and organizational factors, as well as the technologies and how they are used in a system. A walk up this staircase from the technology focused to a more holistic view can take a long time but conscious reflection can help speed the path to improved performance and risk management.

A complete understanding of barrier systems, which is really what risk management is about, requires an understanding of which factors are influencing accident risk, and what can be done to mitigate the risk. This requires that the asset owner thinks not only about “proof testing”, “compliance” or “asset management”, but also about:

  • Leadership
  • Barrier integrity
  • Maintenance
  • Monitoring
  • Design
  • Competence management
  • Permit to work system
  • Dynamics of plant and controls in normal and degraded modes
  • Etc, etc, etc.

In other words – to keep risk under control you need to take the full complexity of your operations into account. A purely technical view on process safety is thus simply not good enough.