Testing is an integral part of operation and maintenance of equipment with a SIL rating. Testing is necessary to ensure the achieved integrity of the safety instrumented function is actually as intended. First of all, the mere assumption of a test frequency  (hours between each proof test, τ) as a direct impact on the calculated probability of failure on demand (PFD) with a given failure rate for dangerous undetected failures, λ_DU :

PFD =  λ_DU x (τ/2)

This function is valid for calculating the average PFD for a single component. For redundant configurations things become more complicated but let us stick to this one for the sake of simplicity. Obviously, if we cut the number of hours between tests in half, we cut the PFD in half. So, the more often we test, the better it is – right? No – Wrong!

Why is that wrong? Testing has two negative sides:

1. It stops production, which means it stops cash flow, which means it costs money there and then
2. It is a source of errors itself, either through increased wear on the system or more likely, a possibility of human error like forgetting to put a system back into automatic mode after testing is done

Of course, this does not mean that we should not test – that is absolutely necessary to make sure the safety function works. Also, over time we can use the results from testing of our functions in the SIS to check whether the assumed failure rates are correct. What it means is – we need to find the right balance between the good side and bad side of testing. In practice, annual testing is often used, and this may be a sweet spot for test frequencies? Sometimes engineers are tempted to increase the test frequency to avoid trouble with PFD numbers after they have bough inferior equipment. People working on the installations tend to strongly oppose this – and rightly so. Buy good components, and test with a reasonable frequency to minimize the impact of the bad things about testing.

Safety critical software must be developed in accordance with certain practices, using specific tools fit for the required level of reliabliity, as well as by an organization with the right competence and maturity for developing such software. These software components are often parts of barrier management systems; bugs may lead to removal of critical functionality that again can lead to an accident situation with physical consequences, such as death, injuries, release of pollutants to the environment and severe material damage.

It is therefore a key question whether such software should be able to run under consumer grade operating systems, not conforming to any reliability development practices? The primary argument why this should be allowed from some vendors is “proven in use”; that they have used the software under said operating system for so many operating hours without incident, such that they feel the system is safe as borne out of experience.

It immedately seems reasonable to put more trust in a system set up that has been field testet over time and shown the expected performance, than a system that has not been tested in the field. Most operarting systems are issued with known bugs in addition to unknown bugs, and a list of bugs will exist for patching. A prioritzation of criticality is made, and the patches are developed accordingly. For Linux systems this patching strategy may be somewhat less organized as development is more distributed and less managed; even a the kernel level. The problem is akin to the classical software security problem; if software with design flaws and bugs is released, any such flaws will be found when a vulnerability is found by externals, or an incident occurs that can show the flaw. The bug or flaw is always inherent in code, and typically stems from lack of good practices during design and code development. In theory, damage resulting from such bugs and flaws shall then be limited by patching the system. In the meantime it is thought that perimeter defences cancounteract the risk of a vulnerability being exploited (this argument may not even hold in the security situation). For bugs affecting the safety in the underlying system, this thinking is flawed because even a single accident may have unacceptable consequences – including loss of human life.

In reliability engineering it is disputed whether a “workflow oriented” or “reliability growth oriented” view on software development and reliability is the most fruitful. Both have their merits. The “ship and patch” thinking inherent in proven in use cases for software indicate a stronger belief in reliability growth concepts. These are models that try to link the number of faults per operating time of the software to duration of discovery period ; most of them are modeled as some form of a Poisson process. It is acknowledged that this is a probabilistic model of deterministic errors inherent in the software, and the stochastic element is whether the actual software states realizing the errors are visited during software execution.

Coming back to operating systems, we see that complexity of such systems have grown very rapidly over the last decades. Looking specifically at the Linux kernel, the development has been tracked over time. The first kernel had about 10.000 lines of code ( in 1990). For the development of kernel version 2.6.29 they added almost 10.000 lines of code per day. If a reliability growth concept is going to work for such a rapid growth in complexity, 10.000 lines of code must be analyzed daily and end up as completely bug-free – and to prove that it is necessary to test every software state for those 10.000 lines of code.

Some research exists to compare effect of coding practices. Microsoft stated in 1992 that they had about 10-20 errors per 1000 lines of code prior to testing and that about 0.5 errors per 1000 lines of code would exist in shipped products.

Compliant development gives no guarantees on flaw and bug-free software. The same goes for development following good security practices – vulnerabilities may still exist. These practices have, however, been developed to minimize the number of design flaws and bugs getting into the shipped product. Structured programming techniques have been shown to produce code with less than 0.1 defects per 1000 lines of code – basically by following a workflow oriented quality regime in tandem with testing. If we assume 0.5 errors per 1000 lines of code in the Linux kernel (the kernel is not the entire OS), we have an estimated 7500 undiscovered bugs in the shipped version of Linux kernel 3.2.

An international rating for security of operating systems exist, the EAL rating. Commercial grade systems have a rating of EAL 4, where as secure RTOS’s tend to be EAL5 (semiformally designed, verified and tested).

The summary seems to be that consumer grade OS’s for life critical automation systems is not the best of ideas – which is why we don’t see too many of them.

When buying technical items with certain requirements – such as reliability requirements / SIL requirements – obtaining documentation showing that the requirements are all met can be a headache. Especially for SIL projects this can be the case, in particular if a particular vendor is not used to providing the correct documentation. When this is the case – what can be done to make sure we do not get delays and problems with non-conformity? The answer is as logical and straight-forward to state, as it is diffcult to implement. Experience has shown that some practices may make obtaining compliance documentation easier – and it is all about communication.

FIrst of all – vendors must be made aware of the requirements at the time they are bidding for the sale – and not only a reference to a standard, but an actual explanation of what it means and what is exepcted. The party selling something should really try to understand this by asking the right questions…. but they don’t always do that.. So, communicating with the vendor from early on is important. This can be done by including requirements in the purchase order or contract.

This is, of course, not enough. So, vendor follow-up should be part of the planning of the engineering activities – just like you would plan to spend resources on requirement setting or participating in FAT’s!

When this has been planned – it is time to step up and help the vendor. Provide a guideline with what they need to deliver of documentation. Provide training to make them understand what is being asked of them – if they do not have the sufficient level of understanding. Make sure the responsible engineer for following up the delivery is also in the know about these requirements. Then the two contacts can speak the same language.

The vendor needs to know what the actual requirements are. It is therefore good practice to develop and supply the safety requirement specification as early as possible. Of course – that may lead to later changes as more information becomes available, but as long as everyone is on board with that and changes are properly managed, including across interfaces, this is a much better situation than requirements coming to vendors too late.

When the vendor has access to the SRS, the follow-up process should be intensified. Ask regular questions about progress – this may be done in an informal manner if the business climate is right. Open up for questions and be a support to the vendor in this process. Keep the conversation going. And make sure you get the progress you need. Tools for “expediting” progress should first and foremost be on the carrot side – let the vendor see real business benefit and value from providing good documentation and fulfilling expectations. Carrots can be things such as better chance of repeat business, improved vendor competence and standing in the market and possibly the option to become a shortlisted or preferred supplier? In addition to a bag of carrots, it may be necessary to carry a stick when soft talking no longer works. Metaphorical sticks may be things like payments tied to documentation deliverables, banning of supplier from future projects and general reputation damage. Hopefully you can keep the sticks hidden in the golf bag.

This provides no guearantee – but thinking about this stuff is infintiely better than not doing anything. My experience is that a little help brings a lot of progress.

To package these thoughts into a more condensed form you may refer to the following infographic – which sums up what to focus on during planning, design and follow-up.