Bears out for honey in the pot: some statistics

This weekend I decided to do a small experiment. Create two virtual machines in the cloud, one running Windows, and one running Linux. The Windows machine exposes RDP (port 3389) to the internet. The Linux machine exposes SSH (port 22). The Windows machines sees more than 10x the brute-force attempts of the Linux machine. 

We capture logs, and watch the logon attempts. Here’s what I wanted to find out: 

  • How many login attempts do we have in 24 hours?
  • What usernames are the bad guys trying with?
  • Where are the attacks coming from?
  • Is there a difference between the two virtual machines in terms of attack frequency?

The VM’s were set up in Azure, so it was easy to instrument them using Microsoft Sentinel. This makes it easy to query the logs and create some simple statistics. 

Where are the bad bears coming from?

Let’s first have a look at the login attempts. Are all the hackers Russian bears, or are they coming from multiple places? 

Windows

On Windows we observed more than 30.000 attempts over 24 hours. The distribution of attacks that the majority came from Germany, then Belarus, followed by Russia and China. We also see that there are some attempts from many countries, all around the globe. 

Logon attempts on a Windows server over 24 hours

Linux

On Linux the situation is similar, although the Chinese bad guys are a lot more intense than the rest of them. We don’t see that massive amount of attacks from Germany on this VM. It is also less popular to attack the Linux VM: only 3000 attempts over 24 hours, about 10% of the number of login attempts observed on the Windows VM. 

Logon attempts on a Linux server over 24 hours

What’s up with all those German hackers?

The German hackers are probably not German, or human hackers. These login attempts are coming from a number of IP addresses known to belong to a known botnet. That is; these are computers in Germany infected with a virus. 

Usernames fancied by brute-force attackers

What are the usernames that attackers are trying to log in with? 

Top 5 usernames on Linux:

Top 5 usernames on Windows: 

We see that “admin” is a popular choice on both servers, which is perhaps not so surprising. On Linux the attackers seem to try a lot of typical service names, for example “ftp” as shown above. Here’s a collection of usernames seen in the logs: 

  • zabbix
  • ftp
  • postgres
  • ansible
  • tomcat
  • git
  • dell
  • oracle1
  • redmine
  • samba
  • elasticsearch
  • apache
  • mysql
  • kafka
  • mongodb
  • sonar

Perhaps it is a good idea to avoid service names as account names, although the username itself is not a protection against unauthorized access. 

There is a lot less of this in the Windows login attempts; here we primarily see variations of “administrator” and “user”. 

Tips for avoiding brute-force attackers

The most obvious way to avoid brute-force attacks from the Internet, is clearly to not put your server on the Internet. There are many design patterns that allow you to avoid exposing RDP or SSH directly on the Internet. For example:

  • Only allow access to your server from the internal network, and set up a VPN solution with multi-factor authentication to get onto the local network remotely
  • Use a bastion host solution, where access to this host is strictly controlled
  • Use an access control solution that gives access through short-lived tokens, requiring multi-factor authentication for token access. Cloud providers have services of this type, such as just-in-time access on Azure or OS Login on GCP.

Firebase: Does serverless mean securityless?

Do you like quizzes or capture the flag (CTF) exercises? Imagine we want to build a platform for creating a capture the flag exercise! We need the platform to present a challenge. When users solve the challenge, they find a “flag”, which can be a secret word or a random string. They should then be able to submit the flag in our CTF platform and check if it is correct or not. 

Red flag
Capture the flag can be fun: looking for a hidden flag whether physically or on a computer

To do this, we need a web server to host the CTF website, and we need a database to store challenges. We also need some functionality to check if we have found the right flag. 

Firebase is a popular collection of serverless services from Google. It offers various easy to use solutions for quickly assembling applications for web or mobile, storing data, messaging, authentication, and so on. If you want to set up a basic web application with authentication and data storage without setting up backends, it is a good choice. Let’s create our CTF proof-of-concept on Firebase using Hosting + Firestore for data storage. Good for us, Google has created very readable documentation for how to add Firebase to web projects.

Firestore is a serverless NoSQL database solution that is part of Firebase. There are basically two ways of accessing the data in Firebase: 

  • Directly from the frontend. The data is protected by Firestore security rules
  • Via an admin SDK meant for use on a server. By default the SDK has full access to everything in Firestore

We don’t want to use a server, so we’ll work with the JavaScript SDK for the frontend. Here are the user stories we want to create: 

  • As an organizer I  want to create a CTF challenge in the platform and store it in Firebase so other users can find it and solve the challenge
  • As a player I want to view a challenge so that 
  • As a player I want to create a form to submit a flag to check that it is correct

We want to avoid using a server, and we are simply using the JavaScript SDK. Diagrams for the user stories are shown below.

User stories
User stories for a simple CTF app example

What about security?

Let’s think about how attackers could abuse the functionalities we are trying to create. 

Story 1: Create a challenge

For the first story, the primary concern is that nobody should be able to overwrite a challenge, including its flag. 

Each challenge gets a unique ID. That part is taken care of by Firestore automatically, so an existing challenge will not be overwritten by coincidence. But the ID is exposed in the frontend, and so is the project metadata. Could an attacker modify an existing record, for example its flag, by sending a “PUT” request to the Firestore REST API?

Let’s say we have decided a user must be authenticated to create a challenge, and implemented this by the following Firebase security rule

match /challenges/{challenges} {
      allow read, write: if request.auth != null;
}

Hacking the challenge: overwriting data

This says nothing about overwriting existing data. It also has no restriction on what data the logged in user has access to – you can both read and write to challenges, as long as you are authenticated. Here’s how we can overwrite data in Firestore using set.

Of course, we need to test that! We have created a simple example app. You need to log in (you can register an account if you want to), and go to this story description page: https://quizman-a9f1b.web.app/challenges/challenge.html#wnhnbjrFFV0O5Bp93mUV

screenshot

This challenge has the title “Fog” and description “on the water”. We want to hack this as another user directly in the Chrome dev tools to change the title to “Smoke”. Let’s first register a new user, cyberhakon+dummy@gmail.com and log in. 

If we open devtools directly, we cannot find Firebase or similar objects in the console. That is because the implementation uses SDV v.9 with browser modules, making the JavaScript objects contained within the module. We therefore need to import the necessary modules ourselves. We’ll first open “view source” and copy the Firebase metadata. 

const firebaseConfig = {
            apiKey: "<key>",
            authDomain: "quizman-a9f1b.firebaseapp.com",
            projectId: "quizman-a9f1b",
            storageBucket: "quizman-a9f1b.appspot.com",
            messagingSenderId: "<id>",
            appId: "<appId>",
            measurementId: "<msmtId>"
        };

We’ll simply paste this into the console while on our target challenge page. Next we need to import Firebase to interact with the data using the SDK. We could use SDK v.8 that is namespaced, but we can stick to v.9 using dynamic imports (works in Chrome although not yet a standard): 

import('https://www.gstatic.com/firebasejs/9.6.1/firebase-app.js').then(m => firebase = m)

and 

import('https://www.gstatic.com/firebasejs/9.6.1/firebase-firestore.js').then(m => firestore = m)

Now firestore and firebase are available in the console. 

First, we initalize the app with var app = firebase.initializeApp(firebaseConfig), and the database with var db  = firestore.getFirestore().  Next we pull information about the challenge we are looking at: 

var mydoc = firestore.doc(db, "challenges", "wnhnbjrFFV0O5Bp93mUV");
var docdata = await firestore.getDoc(mydoc);

This works well. Here’s the data returned: 

  • access: “open”
  • active: true
  • description: “on the water”
  • name: “Fog”
  • owner: “IEiW8lwwCpe5idCgmExLieYiLPq2”
  • score: 5
  • type: “ctf”

That is also as intended, as we want all users to be able to read about the challenges. But we can probably use setDoc as well as getDoc, right? Let’s try to hack the title back to “Smoke” instead of “Fog”. We use the following command in the console: 

var output = await firestore.setDoc(mydoc, {name: “Smoke”},{merge: true})

Note the option “merge: true”. Without this, setDoc would overwrite the entire document. Refreshing the page now yields the intended result for the hacker!

screenshot

Improving the security rules

Obviously this is not good security for a very serious capture-the-flag app. Let’s fix it with better security rules! Our current rules allows anyone who is authenticated to read data, but also to write data. Write here is shorthand for create, update, and delete! That means that anyone who is logged in can also delete a challenge. Let’s make sure that only owner can modify documents. We keep the rule for reading to any logged in user, but change the rule for writing to the following:

Safe rule against malicious overwrite:

allow write: if request.auth != null && request.auth.uid == resource.data.owner;

This means that authenticated users UID must match the “owner” field in the challenge. 

Note that the documentation here shows a method that is not safe – these security rules can be bypassed by any authenticated user: https://firebase.google.com/docs/firestore/security/insecure-rules#content-owner-only

(Read 4 January 2022)

Using the following security rules will allow anyone to create, update and delete data because the field “author_id” can be edited in the request directly. The comparison should be done as shown above, against the existing data for update using resource.data.<field_name>. 

service cloud.firestore {
  match /databases/{database}/documents {
    // Allow only authenticated content owners access
    match /some_collection/{document} {
      allow read, write: if request.auth != null && request.auth.uid == request.resource.data.author_uid
    }
  }
}
// Example from link quoted above

There is, however, a problem with the rule marked “SAFE AGAINST MALICIOUS UPDATES” too; it will deny creation of new challenges! We thus need to split the write condition into two new rules, one for create (for any authenticated user), and another one for update and delete operations. 

The final rules are thus: 

allow read, create: if request.auth != null;
allow update, delete: if request.auth != null && request.auth.uid == resource.data.owner;

Story 2: Read the data for a challenge

When reading data, the primary concern is to avoid that someone gets access to the flag, as that would make it possible for them to cheat in the challenge. Security rules apply to documents, not to fields in a document. This means that we cannot store a “secret” inside a document; access is an all or nothing decision. However, we can create a subcollection within a document, and apply separate rules to that subdocument. We have thus created a data structure like this: 

screenshot of firestore data structure

Security rules are hierarchical, so we need to apply rules to /challenges/{challenge}/private/{document}/ to control access to “private”. Here we want the rules to allow only “create” a document under “private” but not to change it, and also not to read it. The purpose of blocking reading of the “private” documents is to avoid cheating. 

But how can we then compare a player’s suggested flag with the stored one? We can’t in the frontend, and that is the point. We don’t want to expose the data in on the client side. 

Story 3: Serverless functions to the rescue

Because we don’t want to expose the flag from the private subcollection in the frontend, we need a different pattern here. We will use Firebase cloud functions to do that. This is similar to AWS’ lambda functions, just running on GCP/Firebase instead. For our Firestore security, the important aspect is that a cloud function running in the same Firebase project has full access to everything in Firestore, and the security rules do not apply to the admin SDK used in functions. By default a cloud function is assigned an IAM role that gives it this access level. For improved security one can change the roles so that you allow only the access needed for each cloud function (here: read data from Firestore). We haven’t done that here, but this would allow us to improve security even further. 

Serverless security engineering recap

Applications don’t magically secure themselves in the cloud, or by using serverless. With serverless computing, we are leaving all the infrastructure security to the cloud provider, but we still need to take care of our workload security. 

In this post we looked at access control for the database part of a simple serverless web application. The authorization is implemented using security rules. These rules can be made very detailed, but it is important to test them thoroughly. Misconfigured security rules can suddenly allow an attacker to bypass your intended control. 

Using Firebase, it is not obvious from the Firebase Console how to set up good application security monitoring and logging. Of course, that is equally important when using serverless as other types of infrastructure, both for detecting attacks, and for forensics after a successful breach. You can set up monitoring Google Cloud Monitoring for Firebase resources, including alerts for events you want to react to. 

As always: basic security principles still hold with serverless computing!

How to discover cyberattacks in time to avoid disaster

Without IT systems, modern life does not exist. Whether we are turning on our dishwasher, ordering food, working, watching a movie or even just turning the lights on, we are using computers and networked services. The connection of everything in networks brings a lot of benefits – but can also make the infrastructure of modern societies very fragile.

Bus stop with networked information boards
Cyber attacks can make society stop. That’s why we need to stop the attackers early in the kill-chain!

When society moved online, so did crime. We hear a lot about nation state attacks, industrial espionage and cyber warfare, but for most of us, the biggest threat is from criminals trying to make money through theft, extortion and fraud. The difference from earlier times is that we are not only exposed to “neighborhood crime”; through the Internet we are exposed to criminal activities executed by criminals anywhere on the planet. 

We see in media reports every week that organizations are attacked. They cause serious disruptions, and the cost can be very high. Over Christmas we had several attacks happening in Norway that had real consequences for many people: 

  • Nortura, a cooperative owned by Norwegian farmers that processes meat and eggs, was hit by a cyber attack that made them take many systems offline. Farmers could not deliver animals for slaughtering, and distribution of meat products to stores was halted. This is still the situation on 2 January 2022, and the attack was made public on 21 December 2021. 
  • Amedia, a media group publishing mostly local newspapers, was hit with ransomware. The next 3 days following the attack on the 28th of December, most of Amedia’s print publications did not come out. They managed to publish some print publications through collaboration with other media houses. 
  • Nordland fylkeskommune (a “fylkeskomune” is a regional administrative level between municapalities and the central government in Norway) and was hit with an attack on 23 December 2021. Several computer systems used by the educational sector have been taken offline, according to media reports. 

None of the reports mentioned above have been linked to the log4shell chaos just before Christmas. The Amedia incident has been linked to the Printer Nightmare vulnerability that caused panic earlier in 2021. 

What is clear is that there is a serious and very real threat to companies from cyber attacks – affecting not only the companies directly, but entire supply chains. The three attacks mentioned above all happened within two weeks. They caused disruption of people’s work, education and media consumption. When society is this dependent on IT systems, attacks like these have real consequences, not only financially, but to quality of life.

This blog post is about how we can improve our chances of detecting attacks before data is exfiltrated, before data is encrypted, before supply chains collapse and consumers get angry. We are not discussing what actions to take when you detect the attack, but we will assume you already have an incident response plan in place.

How an attack works and what you can detect

Let’s first look at a common model for cyber attacks that fit quite well with how ransomware operators execute; the cyber kill-chain. An attacker has to go through certain phases to succeed with an attack. The kill-chain model uses 7 phases. A real attack will jump a bit back and forth throughout the kill-chain but for discussing what happens during the attack, it is a useful mental model. In the below table, the phases marked in yellow will often produce detectable artefacts that can trigger incident response activities. The blue phases of weaponization (usually not detectable) and command & control, actions on objectives (too late for early response) are not discussed further. 

Defensive thinking using a multi-phase attack model should take into account that the earlier you can detect an attack, the more likely you are to avoid the most negative consequences. At the same time, the earlier phases give you detections with more uncertainty, so you do risk initiating response activities that can hurt performance based on false positives. 

Detecting reconnaissance

Before the attack happens, the attacker will identify the victim. For some attacks this is completely automated but serious ransomware attacks are typically human directed and will involve some human decision making. In any case, the attacker will need to know some things about your company to attack it. 

  • What domain names and URL’s exist?
  • What technologies are used, preferably with version numbers so that we can identify potential vulnerabilities to exploit?
  • Who are the people working there, what access levels do they have and what are their email addresses? This can be used for social engineering attacks, e.g. delivering malicious payloads over email. 

These activities are often difficult to detect. Port scans and vulnerability scans of internet exposed infrastructure are likely to drown in a high number of automated scans. Most of these are happening all the time and not worth spending time reacting to. However, if you see unusual payloads attempted, more thorough scans, or very targeted scans on specific systems, this can be a signal that an attacker is footprinting a target. 

Other information useful to assess the risk of an imminent attack would be news about attacks on other companies in the same business sector, or in the same value chain. Also attacks on companies using similar technical set-ups (software solutions, infrastructure providers, etc) could be an early warning signal. In this case, it would be useful to prepare a list of “indicators of compromise” based on those news reports to use for threat hunting, or simply creating new alerts in monitoring systems. 

Another common indicator that footprinting is taking place, is an increase in the number of social engineering attempts to elicit information. This can be over phone, email, privacy requests, responses to job ads, or even in person. If such requests seem unusual it would be useful for security if people would report it so that trends or changes in frequency can be detected. 

Detecting delivery of malicious payloads

If you can detect that a malicious payload is being delivered you are in a good position to stop the attack early enough to avoid large scale damage. The obvious detections would include spam filters, antivirus alerts and user reports. The most common initial intrusion attack vectors include the following: 

  • Phishing emails (by far the most common)
  • Brute-force attacks on exposed systems with weak authentication systems
  • Exploitation of vulnerable systems exposed to the Internet

Phishing is by far the most common attack vector. If you can reliably detect and stop phishing attacks, you are reducing your risk level significantly. Phishing attacks usually take one of the following forms: 

  • Phishing for credentials. This can be username/password based single-factor authentication, but also systems protected by MFA can be attacked with easy to use phishing kits. 
  • Delivery of attachments with malicious macros, typically Microsoft Office. 
  • Other types of attachments with executable code, that the user is instructed to execute in some way

Phishing attacks create a number of artefacts. The first is the spam filter and endpoint protection; the attacker may need several attempts to get the payload past first-line of defences. If there is an increase in phishing detections from automated systems, be prepared for other attacks to slip past the filters. 

The next reliable detection is a well-trained workforce. If people are trained to report social engineering attempts and there is an easy way to do this, user reports can be a very good indicator of attacks. 

For brute-force attacks, the priority should be to avoid exposing vulnerable systems. In addition, creating alerts on brute-force type attacks on all exposed interfaces would be a good idea. It is then especially important to create alerts to successful attempts. 

For web applications, both application logs and WEF logs can be useful. The main problem is again to recognize the attacks you need to worry about in all the noise from automated scans. 

Detecting exploitation of vulnerabilities

Attackers are very interested in vulnerabilities that give them access to “arbitrary code execution”, as well as “escalation of privileges”. Such vulnerabilities will allow the attacker to install malware and other tools on the computer targeted, and take full control over it. The primary security control to avoid this, is good asset management and patch management. Most attacks are exploiting vulnerabilities where a patch exists. This works because many organizations are quite late at patching systems. 

A primary defense against exploitation of known vulnerabilities is endpoint protection systems, including anti-virus. Payloads that exploit the vulnerabilities are recognized by security companies and signatures are pushed to antivirus systems. Such detections should thus be treated as serious threat detections. It is all well and good that Server A was running antivirus that stopped the attack, but what if virus definitions were out of date on Server B?

Another important detection source here would be the system’s audit logs. Did the exploitation create any unusual processes? Did it create files? Did it change permissions on a folder? Log events like this should be forwarded to a tamper proof location using a suitable log forwarding solution. 

To detect exploitation based on log collection, you will be most successful if you are focusing on known vulnerabilities where you can establish patterns that will be recognizable. For example, if you know you have vulnerable systems that cannot be patched for some reason, establishing specific detections for exploitation can be very very valuable. For tips on log forwarding for intrusion detection in Windows, Microsoft has issued specific policy recommendations here

Detecting installation (persistence)

Detecting installation for persistence is usually quite easy using native audit logs. Whenever a new service is created, a scheduled task is created, or a cron job, an audit log entry should be made. Make sure this is configured across all assets. Other automated execution patterns should also be audited, for example autorun keys in the Windows registry, or programs set to start automatically when a user logs in.

Another important aspect is to check for new user accounts on the system. Account creation should thus also be logged. 

On web servers, installation of web shells should be detected. Web shells can be hard to detect. Because of this, it is a good idea to monitor file integrity of the files on the server, so that a new or changed file would be detected. This can be done using endpoint protection systems.

How can we use these logs to stop attackers?

Detecting attackers will not help you unless you take action. You need to have an incident response plan with relevant playbooks for handling detections. This post is already long enough, but just like the attack, you can look at the incident response plan as a multi-phase activity. The plan should cover: 

  1. Preparations (such as setting up logs and making responsibilities clear)
  2. Detection (how to detect)
  3. Analysis and triage (how to classify detections into incidents, and to trigger a formal response, securing forensic evidence)
  4. Containment (stop the spread, limit the blast radius)
  5. Eradication (remove the infection)
  6. Recovery (recover the systems, test the recovery)
  7. Lessons learned (how to improve response capability for the next attack, who should we share it with)

You want to know how to deal with the data you have collected to decide whether this is a problem or not, and what to do next. Depending on the access you have on endpoints and whether production disturbances during incident response are acceptable, you can automate part of the response. 

The most important takeaway is:

Set up enough sensors to detect attacks early. Plan what to do when attacks are detected, and document it. Perform training exercises on simulated attacks, even if it is only tabletop exercises. This will put you in a much better position to avoid disaster when the bad guys attack! 

Things I learned from starting – and shutting down – a company

In 2016 I worked as a business development manager at Lloyd’s Register‘s consulting unit in Norway. We were building up a new service within industrial cybersecurity, and had a few good people on the team. We had great plans but then difficult times in the oil and gas sector started to cause problems for us. The order books were close to empty and the company started offering severance packages. We lost two key resources for our cybersecurity project, and internal funding for “future growth” was hard to obtain in this economic climate. That was the birth of the company Cybehave.

Starting a company with little sense of direction

We started Cybehave first a development project where we wanted to automate cybersecurity risk assessment, to make such services available to smaller companies. We got seed funding from Innovation Norway – called “markedsavklaringsstøtte” (Norwegian for “market validation grant”), about NOK 85.000. We also got a free workplace for a while at a startup incubator while establishing a “minimum viable product”. A key problem was that we didn’t really know what was a viable product, or who the customers were. We were searching for pilot customers, looking at small and medium sized businesses. All our real-world sales experience, however, was from LR. We were used to working with global energy companies, government agencies and international manufacturing organizations. The contacts we had, and the typical way to initiate conversations in that space, was irrelevant in the SMB space. So we were to a large degree guessing what those SMB’s would need in terms of security, having problems agreeing between ourselves what exactly our value proposition was. While doing this, our laid-off PhD level expert in risk management was building a minimum viable product by coding up a web application in Django/Python.

Without a clear understanding of your market, it hard to know what focus on.

We did focus groups, where we invited companies from many sectors. We got little useful feedback. Vi visited a lot of companies, trying to convince them that they needed cybersecurity risk management and awareness training. They were not particularly interested, and our message was perhaps not very clear either.

Before you invest a lot of time (and money) in your product, know who the customer is, and what problem you are solving for them. If you don’t know, spend time searching for a problem to solve instead of a customer who has the problem you have imagined must be important to others.

Without money, life is hard

Still without customers, we wanted to sell our great approach to human centric cybersecurity. We were thinking that “we don’t have customers because we don’t have money for marketing”. Because of this, we wanted a bring an investor on board. One of the co-founders focused a lot on this, but finding an investor who is interested without customers and cash flow, and without a very clear value proposition was difficult, for some reason. Here’s what we learned:

  • Local angel investors want a lot for equity without contributing much money. They have limited networks and understanding of B2B markets.
  • Pitching in start-up events requires to have a really good story. B2C stories tend to win over B2B stories, at least if your story isn’t particularly exiting
  • Financial estimates have very little value in the early phase. They are mostly baseless guesstimates, sprinkled with wishful thinking.
  • Professional investors give a lot of very useful feedback. Talking to investment funds even if you are not in a place where you would be a good investment. You learn how they think, what they are looking for: clarity of benefit provided, growth potential, intellectual property rights, and capabilities of the management team/founders.

End of story, we did not get any external investment. The story was a bit too vague to compete with B2C for small-scale investors – or for the offers we did get we were too greedy to say “yes” to give away too much ownership, and too early to be interesting to equity funds.

We went to the government again – Innovation Norway. They granted us a “commercialisation grant” of NOK 450.000. We received the first pay-out early 2019, 50% of the grant. That process was not without effort, but with a better story to tell, a better plan, and a working prototype to demonstrate part of we wanted to do was enough to get that money. And it was a nice grant because we did not have to give away equity – although the amount of money was not anywhere close to what we wanted to get sufficient growth. Because of this, and our not so successful attempts at getting investors on board, we switched the strategy of getting funding the old-fashioned way; through positive cash flow.

Because the company was not making money, and we did not have any serious funding in place, nobody was working on the project full-time. We all had day jobs, and demanding day jobs at that. Building up a security team at a global IT company, leading a department at a regional hospital. This further hampered product development.

You need a realistic funding plan from the beginning. Think through what you want from an external investor (money – or network, operational experience, support in addition) and how much of the equity (and control) you are willing to part with.

We did not want to make money by selling consulting hours. We wanted to build a scalable alternative. However, to provide cash flow to the company, we decided to start doing some consulting. However, doing that on top of a day job that had to be followed up, did not leave much time for building those scalable services!

Create a realistic plan for input resources, whether time or money. Full time work on the side of bringing in money for development through consulting is not a sustainable model.

Administration requires work too

It is easy to focus on the customer, the big ideas, developing software (more about that later). if you don’t keep up with administrative needs, there will be problems.

Accounting is important. There are many software companies selling “do-it-yourself” accounting solutions. Unless you enjoy accounting and actually know what you are doing, avoid the DIY solutions. IT is hard to know which account to use for a certain expense, and what services bought outside the country that should be reported for VAT or not. You could spend time learning all this, but unless that is your core business or you enjoy the details of accounting, get help. Top three accounting tips must be:

  • Engage an accountant.
  • Set up integrations between your bank accounts and your accounting system.
  • Use the accounting data to keep track of your company’s finances. Set up dashboards or reports that make sense to you. As a bare minimum you should get monthly statements on cash flow, liquidity and expenses in key categories (e.g. cloud computing, travel, salary).

In addition to accounting you will need to report regularly to the government. In Norway you will have to create a VAT tax report every other month. Failing to report on time will cause trouble – or fines from the tax authorities. This job is definitely best left to an accountant again! The same goes for the annual accounts and shareholder registry if your company is a limited company with shares.

Get an accountant, and set up bank integration solutions and automation as early as you can. This will free up a lot of time and worry so you can focus on building your company.

A successful product: PrivacyBox

In 2018 I worked at Sportradar as my full-time day job. There I met the data protection officer, newly hired, who was trying to get this multinational company in shape for the GDPR. Together we created an internal tool for a personal data inventory solution. We also saw that there were a lot of challenges related to management of requests from data subjects. The most common solution was to publish an e-mail address on the privacy policy page where people could submit requests for access to data, deletion or other rights they want to exercise under the GDPR or other policies. We agreed to take my colleague from Sportradar on as a shareholder in Cybehave and to develop a good solution for handling privacy rights. The counterpart at Sportradar was the head of legal, to avoid conflicts of interest. Sportradar would be a pilot customer, with free access the first months (before the product was actually very usable) as long as we got feedback on the software. Then they would get a discount for some time before the price goes up to the market price.

This gave us a very different situation from the security awareness and risk solution: someone with actual use for the product who could tell us what they needed. It was mainly I who developed the first version of this software, as a prototype. We got a lot of great features in, and the customer was happy with the product. It was in use by Sportradar globally across all their brands from 2019 to 31 December 2021. They had to switch vendor because Cybehave is being dissolved but they were happy with the solution.

  • Have a pilot customer before you write any type of code
  • The pilot customer should have a clear need to satisfy and opinions on how the system should work
  • The pilot customer should have sufficient volume of work to be done in the software that you get real-world experience and feedback

In addition to the help we got from the clear feedback from the pilot customer, we quickly learned a few other things:

  • Create great end-user documentation that tells users how to accomplish tasks.
  • For “one-time users” such as data subjects making requests, make filling in the form as quick and easy as possible
  • Solutions that filter spam are important when publishing forms online on pages with high-volume traffic. An e-mail with a confirmation link is a simple and effective solution for this.
  • Application logging is extremely important for troubleshooting and customer support requests
  • Be prepared to answer customer support requests quickly. Keeping the customer happy means making sure they can get their work done, even when the software solution has a bug or is missing a feature

Work closely with a pilot customer to create a product that actually solves a problem. Remember that documentation, logging and support are essential parts of the service offering!

Don’t develop software alone

Cybehave was a company without full-time employees. In fact, most of the time it did not have any employees at all. In the beginning, the first prototype of a SaaS software was created by the colleague that was let go from LR. She was a brilliant risk analyst, and great at scientific computing. That does not make you a software engineer. The majority, however, was written by me. The other two co-founders were non-technical and did not write code. Not sure I am brilliant at anything, but I am also not a software engineer by education. I did, however, learn a great deal from the Cybehave project, as well as from working at Sportradar. Key take-aways for the next time:

  1. Don’t write software alone. It is too much work and too easy to make serious mistakes leading to vulnerabilities and nasty bugs.
  2. Spend more time thinking about architecture and design patterns than actually writing code.
  3. Iterate. When your new feature works, it is not done. Work on it until it becomes good – think about and measure performance, reliability, user experience. And most of all: get outside feedback on how well it works – it will all be easy to you, since you created it!
  4. Test. Because of the lack of formal software engineering background and a focus on “creating the things” as a one-man show, not much testing was done when writing Cybehave’s software. Testing is extremely important for both performance and security.
  5. Don’t create features the customer does not need. All software will need to be maintained, the less code you have, the less interest there will be to pay on the technical debt.

For PrivacyBox we sometimes needed to improve features or add new ones. At one point, we decided to hire a freelancer to do some improvements. That freelancer was a professional software engineer who was not necessarily cheap per hour, but created high-quality code, improved architecture and provided very helpful feedback on technical details. If your team does not have the competence needed and you cannot afford to hire someone, contract with good freelancers for specific tasks and make sure to work closely with them.

Automation and Git hygiene provide a lot of value

You should not make software development a solo project, and testing is important. If you are a non-technical founder but your company makes software, make sure to talk to your technical team about how to ensure good quality of the software you produce. Even with a small team, or with freelancers on board for specific features, you will gain a lot by setting up automated tests and build pipelines. This will reduce the number of bugs and provide help to build better software.

  • Set up at least three branches in Git: development, test, production
  • Push often to development to make sure you do not lose work
  • Use feature branches that will merge to development
  • Merge to test branch should automatically run important tests. Those should include static analysis and software component analysis as a minimum. You should also have unit tests and integration tests running in a software test suite. If tests fail, you should not be able to merge the branch into production.
  • When you merge to production, your pipeline should automatically push the changes to the production servers. Most likely you will be running your software on public cloud infrastructure. Public cloud providers will typically have good documentation available for how to set up CI/CD pipelines.

Application security bare minimum practices

Nothing will erode your customer’s trust as fast as a compromised software solution. Security is business critical, not only to you but also to your customer. Because of this, you should make sure that the software you create follows some key practices.

  1. Ensure identity and authorization management is properly implemented. Use single sign-on solutions for B2B interactions when possible. If you implement your own authentication and authorization system, make sure passwords are strong enough, hashed and salted, and that multifactor authentication is available and possibly required.
  2. Log all security events and create alerts for unexpected events. Important events include authentication, password change, privilege escalation (if multiple authorization levels exist), user creation, unauthorized access/transaction attempts, all privileged access/transactions. In addition, there may be context specific events that are important to track, such as data deletion, data sharing, etc.
  3. Ensure input validation is applied for all user generated input. This also applies to responses from third-party API’s.
  4. Make sure there are no secrets in your code. Secrets should be injected at run-time and be possible to rotate.

Follow good software engineering practices form the start. If you don’t you will get a lot of technical debt, which means there will be so much maintenance to do that you will never catch up.

Lessons from shutting it down

So, Cybehave came to an end. Closing down a software company, means shutting down a lot of services. It would have been much easier to do this, if we had an inventory of all online services and software solutions we were using. When starting to shut down our operations, we had to create this inventory. Here are some categories of services we were using:

  • Transactional mail providers
  • Cloud services (we were running on Google cloud IaaS and PaaS solutions)
  • Office/collaboration software
  • CRM and marketing solutions
  • Github organization with private and public repositories
  • Accounting software
  • Mobile apps from banks, etc.

Just keeping track of the online accounts and services used in a spreadsheet would be a great help. I noticed that we had accounts with many SaaS providers that we were not using; we had simply tried them out and left the accounts active when abandoning them. With a cloud software inventory and a practice to shut down unused accounts we would not only make it easier to shut down the company, we would also have reduced our attack surface.

Shutting down a company also means reporting this to the authorities. We got good help from our accountant in doing this, which takes way the uncertainty about what is required.

Telling our customers has also been important, of course. This should be done in good time, so the customers can transfer data and systems to new solutions if your products are being discontinued. We see that this requires a bit of support time and extra engineering effort to create good data transfer solutions. Factoring in the time to do this is important so that no bridges are burned and contractual obligations are met.

If you are shutting down your company, set aside enough time for technical shutdowns, mandatory reporting, and most importantly, taking care of your business relationships.

Postludium

Cybehave has been a great journey, and the company was actually profitable most years. Most of the cash flow came from consulting, where we have had the privilege of helping software companies, healthcare authorities, municipalities, construction companies. As much as I enjoy creating software, working directly with customers creating value and improving security is the real motivator. Today I am back in a great company where I can do this every day – with real positive impact.

Talking to fund managers and potential customers that never resulted in investments or sales have also been interesting. Start-up life is full of contrasts, at one point we were sitting in a meeting with top management of a multinational engineering company one day, and meeting a potential customer the next day where all 6 employees shared one office that was filled with cardboard boxes and laptops on the floor. It is rewarding but also tiring. But without sufficient financial muscles, the impact you want to make will remain a dream.

Although leaving a project you have put thousands of hours into will inevitably make you feel a bit melancholic, the future is bright, exiting and fast paced!

Happy new year – cybersecurity is still the main focus for 2022. Working to keep the lights on, hospitals running and supply chains safe from hackers.

My new project is working for DNV Cybersecurity, where we are building the world’s best industrial cybersecurity provider within DNV’s Accelerator. DNV’s purpose is to safeguard life, property and the environment – which is very close to heart for me. DNV Cybersecurity has recently joined forces Applied Risk, and this is only the beginning. I am therefore looking forward to making impact together with great colleagues at DNV, where fast growth will allow us to bring the best security solutions for the real world to more customers around the world, defending hospitals, the power grid, shipping, food supply chains and the energy markets from hackers also in 2022.

Happy new year to all of you!

Vendor Security Management: how to decide if tech is safe (enough) to use

tl;dr: Miessler is right. We need to focus on our own risk exposure, not vendor security questionnaires

If you want to make a cybersecurity expert shiver, utter the words “supply chain vulnerabilities”. Everything we do today, depends on a complex mixture of systems, companies, technologies and individuals. Any part of that chain of interconnected parts can be the dreaded weakest link. If hackers can find that weak link, the whole house of cards comes crumbling down. Managing cyber supply chain risk is challenging, to say the least. 

Most companies that have implemented a vendor cybersecurity risk process, will make decisions based on a questionnaire sent to the vendor during selection. In addition, audit reports for recognized standards such as ISO 27001, or SOC2, may be shared by the company and used to assess the risk. Is this process effective at stopping cyberattacks through third parties? That is at least up for debate.

Daniel Miessler recently wrote a blog post titled It’s time for vendor security 2.0, where he argues that the current approach is not effective, and that we need to change the way we manage vendor risks. Considering how many cybersecurity questionnaires Equifax, British Airways and Codecov must have filled in before being breached, it is not hard to agree with @danielmiessler about this. What he argues in his blog is: 

  1. Cybersecurity reputation service (rating companies, etc) are mostly operating like the mob, and security questions are mostly security theater. None of this will save you from cyber armageddon.
  2. Stay away from companies that seem extremely immature in terms of security
  3. Assume the vendor is breached
  4. Focus more on risk assessment under the assumption that the vendor is breached than questionable questionnaires. Build threat models and mitigation plans, make those risks visible. 

Will Miessler’s security 2.0 improve things?

Let’s pick at the 4 numbered points above one by one. 

Are rating companies mobsters? 

There are many cybersecurity rating companies out there. They take measure of themselves to be the Moody’s or S&P’s of cybersecurity. The way they operate is they pull in “open source information about cybersecurity posture” of companies. They also say that they enrich this information with other data that only they have access to (that is, they buy data from marketing information brokers and perform data exchange with insurance companies). Then they correlate this information in more or less sound statistical ways (combined with a good dose of something called expert judgment – or guessing, as we can also call it) with known data breaches and create a security score. Then they claim that using companies with a bad score is dangerous, and with a good score is much better. 

This is definitely not an exact science, but it does seem reasonable to assume that companies that show a lot of poor practice such as a lack of patching, botnet infected computers pinging out to sinkholes and so on, have worse security management than similar companies that do not have these indicators. Personally, I think a service like this can help sort the terrible ones from the reasonably OK ones. 

Then, are they acting as mobsters? Are they telling you “we know about all these vulnerabilities, if you don’t pay us we will tell your customers?”. Not exactly. They are telling everyone willing to pay for access to their data these things, but they are not telling you about it, unless pay them. It is not exactly in line with accepted standards of “responsible disclosure”. At the same time, their findings are often quite basic and anyone bothering to look could find the same things (such as support for old ciphers on TLS or web servers leaking use of an old PHP version). Bottom line, I think their business model is acceptable and that the service can provide efficiency gains for a risk assessment process. I agree with Miessler that trusting this to be a linear scale of cyber goodness is naive at best, but I do think companies with a very poor security rating would be more risky to use than those with good ratings. 

mobster planning his next security rating extortion of SaaS cybersecurity vendors
Some security vendors have a business model that resemble extortion rackets of a 1930’s mobster. But even mobsters can be useful at times.

Verdict – usefulness: rating services can provide a welcome substitute or addition for slower ways of assessing security posture. An added benefit is the ability to see how things develop over time. Small changes are likely to be of little significance, but a steady improvement of security rating over time is a good sign. These services can be quite costly, so it is worth thinking about how much money you want to throw at it. 

Verdict – are they mobsters? They are not mobsters but they are also not your best friends. 

Are security questionnaires just security theater? 

According to Miessler, you should slim down your security questionnaires to two questions: 

  1. “when was the last time you were breached (what happened, why, and how did you adjust)”?, 
  2. and “do you have security leadership and a security program?”.

The purpose of these questions is to judge if they have a reasonable approach to security. It is easy for people to lie on detailed but generic security forms, and they provide little value. To discover if a company is a metaphorical “axe murderer” the two questions above are enough, argues Miessler. He may have a point. Take for example a typical security questionnaire favorite: “does your company use firewalls to safeguard computers from online attacks?” Everyone will answer “yes”. Does that change our knowledge about their likelihood of being hacked? Not one bit. 

Of course, lying on a short questionnaire with Miessler’s 2 questions is not more difficult than lying on a long and detailed questionnaire. Most companies would not admit anything on a questionnaire like this, that is not already publicly known. It is like flying to the US a few years ago where they made you fill out an immigration questionnaire with questions like “are you a terrorist?” and “have you been a guard at a Nazi concentration camp during WWII”. It is thus a good question if we can even just scrap the whole questionnaire. If the vendor you are considering is a software firm, at least if it is a “Software as a Service” or another type of cloud service provider, they are likely to have some generic information about security on their web page. Looking up that will usually be just as informative as any answer to the question above. 

Verdict: Security questionnaires are mostly useless – here I agree with Miessler. I think you can even drop the minimalist axe murderer detection variant, as people who lie on long forms probably lie on short forms too. Perhaps a good middle ground is to first check the website of the vendor for a reasonable security program description, and if you don’t see anything, then you can ask the two questions above as a substitute. 

Stay away from extremely bad practice

Staying away from companies with extremely bad practice is a good idea. Sometimes this is hard to do because business needs a certain service, and all potential providers are horrible at security. But if you have a choice between someone with obviously terrible security habits and someone with a less worrying security posture, this is clearly good advice. Good ways to check for red flags include: 

  • Create a user account and check password policies, reset, etc. Many companies allow you to create free trial accounts, which is good for evaluating security practices as well. 
  • Check if the applications are using outdated practices, poor configuration etc. 
  • Run sslscan to check if they are vulnerable to very old crypto vulnerabilities. This is a good indicator that patching isn’t exactly a priority.

Verdict: obviously a good idea.

Assume the vendor is breached and create a risk assessment

This turns to focus on your own assets and risk exposure. Assuming the vendor is breached is obviously a realistic start. Focusing on how that affects the business and what you can do about it, makes the vendor risk assessment about business risk, instead of technical details that feel irrelevant. 

Miessler recommends: 

  • Understand how the external service integrates into the business
  • Figure out what can go wrong
  • Decide what you can do to mitigate that risk

This is actionable and practical. The first part here is very important, and to a large degree determines how much effort it is worth putting into the vendor assessment. If the vendor will be used for a very limited purpose that does not involve critical data or systems, a breach would probably not have any severe consequences. That seems acceptable without doing much about it. 

On the other hand, what if the vendor is a customer relationship management provider (CRM), that will integrate with your company’s e-commerce solution, payment portal, online banking and accounting systems? A breach of that system could obviously have severe consequences for the company in terms of cost, reputation and legal liabilities. In such a case, modeling what could happen, how one can reduce the risk and assessing whether the residual risk is acceptable would be the next steps.

Shared responsibility – not only in the cloud

Cloud providers talk a lot about the shared responsibility model (AWS version). The responsibility for security of software and data in the cloud is shared between the cloud provider and the cloud customer. They have documentation on what they will take care of, as well as what you as a customer need to secure yourself. For the work that is your responsibility, the cloud provider will typically give you lots of advice on good practices. This is a reasonable model for managing security across organizational interfaces – and one we should adopt with other business relationships too. 

The most mature software vendors will already work like this, they have descriptions of their own security practices that you can read. They also have advice on how you should set up integrations to stay secure. The less mature ones will lack both the transparency and the guidance. 

This does not necessarily mean you should stay away from them (unless they are very bad or using them would increase the risk in unacceptable ways). It means you should work with them to find good risk mitigations across organizational interfaces. Some of the work has to be done by them, some by you. Bringing the shared responsibility for security into contracts across your entire value chain will help grow security maturity in the market as a whole, and benefit everyone. 

Questionnaires are mostly useless – but transparency and shared responsibility is not. 

In Miessler’s vendor security 2.0 post there is a question about what vendor security 3.0 will look like. I think that is when we have transparency and shared responsibility established across our entire value chain. Reaching this cybersecurity Nirvana of resilience will be a long journey – but every journey starts with a first step. That first step is to turn the focus on how you integrate with vendors and how you manage the risk of this integration – and that is a step we can take today. 

Application security in Django projects

This is a quick blog post on how to remove some typical vulnerabilities in Django projects.

coffee shop interior
Even a coffee shop visitor registration app needs to take app security into account

The key aspects we are looking at are:

  • Threat modeling: thinking through what attackers could do
  • Secrets management with dotenv
  • Writing unit tests based on a threat model
  • Checking your dependencies with safety
  • Running static analysis with bandit

Threat model

The app we will use as an example here is a visitor registration app to help restaurants and bars with COVID-19 tracing. The app has the following key users:

  • SaaS administrator: access to funn administration of the app for multiple customers (restaurants and bars)
  • Location administrator: access to visitor lists for individual location
  • Personal user: register visits at participating locations, view their own visit history and control privacy settings
  • Unregistered user: register visits at participating locations, persistent browser session lasting 14 days

The source code for this app is available here: https://github.com/hakdo/besokslogg/.

We use the keywords of the STRIDE method to come up with quick attack scenarios and testable controls. Note that most of these controls will only be testable with custom tests for the application logic.

Attack typeScenarioTestable controls
SpoofingAttacker guesses passwordPassword strength requirement by Django (OK – framework code)

Lockout after 10 wrong consecutive attempts (need to implement in own code) (UNIT)
Tampering
RepudiationAttacker can claim not to have downloaded CSV file of all user visits.CSV export generates log that is not readable with the application user (UNIT)
Information disclosureAttacker abusing lack of access control to gain access to visitor list

Attacker steals cookie with MitM attack

Attacker steals cookie in XSS attack
Test that visitor lists cannot be accessed from view without being logged in as the dedicated service account. (UNIT)

Test that cookies are set with secure flag. (UNIT OR STATIC)

Test that cookies are set with HTTPOnly flag. (UNIT or STATIC)

Test that there are no unsafe injections in templates (STATIC)
Denial of serviceAttacker finds a parameter injection that crashes the applicationCheck that invalid parameters lead to a handled exception (cookies, form inputs, url parameters)
Elevation of privilegeAttacker gains SaaS administrator access through phishing.Check that SaaS administrator login requires a safe MFA pattern (UNIT or MANUAL)
Simple threat model for contact tracing app

Secrets management

Django projects get a lot of their settings from a settings.py file. This file includes sensitive information by default, such as a SECRET_KEY used to generate session cookies or sign web tokens, email configurations and so on. Obviously we don’t want to leak this information. Using python-dotenv is a practical way to deal with this. This package allows you to include a .env file with your secrets as environment variables, and then to include then into settings using os.getenv(‘name_of_variable’). This way the settings.py file will not contain any secrets. Remember to add your .env file to .gitignore to avoid pushing it to a repository. In addition, you should use different values for your development and production environment of all secrets.

from dotenv import load_dotenv
load_dotenv()

SECRET_KEY = os.environ.get('SECRET_KEY')

In the code snippet above, we see that SECRET_KEY is no longer exposed. Use the same technique for email server configuration and other sensitive data.

When deploying to production you need to set the environment variables in that environment using a suitable and secure manner to do it. You should avoid storing configurations in files on the server.

Unit tests

As we saw in the threat model, the typical way to fix a security issue is very similar to the typical way you would fix a bug.

  1. Identify the problem
  2. Identify a control that solves the problem
  3. Define a test case
  4. Implement the test and develop the control

In the visitor registration app, an issue we want to avoid is leaking visitor lists for a location. A control that avoids this is an authorisation check in the view that shows the visitor list. Here’s that code.

@login_required()
def visitorlist(request):
    alertmsg = ''
    try:
        thislocation = Location.objects.filter(service_account = request.user)[0]
        if thislocation:
            visits = Visit.objects.filter(location = thislocation).order_by('-arrival')
            chkdate = request.GET.get("chkdate", "")
            if chkdate:
                mydate = datetime.datetime.strptime(chkdate, "%Y-%m-%d")
                endtime = mydate + datetime.timedelta(days=1)
                visits = Visit.objects.filter(location = thislocation, arrival__gte=mydate, arrival__lte=endtime).order_by('-arrival')
                alertmsg = "Viser besøkende for " + mydate.strftime("%d.%m.%Y")
            return render(request, 'visitor/visitorlist.html', {'visits': visits, 'alertmsg': alertmsg})
    except:
        print('Visitor list failed - wrong service account or no service account')
        return redirect('logout')

Here we see that we first require the user to be logged in to visit this view, and then on line 5 we check to see if we have a location where the currently logged in user is registered as a service account. A service account in this app is what we called a “location administrator” in our role descriptions in the beginning of our blog post. It seems our code already implements the required security controls, but to prove that and to make sure we detect it if someone changes that code, we need to write a unit test.

We have written a test where we have 3 users created in the test suite.

class VisitorListAuthorizationTest(TestCase):

    def setUp(self):
        # Create three users
        user1= User.objects.create(username="user1", password="donkeykong2016")
        user2= User.objects.create(username="user2", password="donkeykong2017")
        user3= User.objects.create(username="user3", password="donkeykong2018")
        user1.save()
        user2.save()

        # Create two locations with assigned service accounts
        location1 = Location.objects.create(service_account=user1)
        location2 = Location.objects.create(service_account=user2)
        location1.save()
        location2.save()
    
    def test_return_code_for_user3_on_visitorlist_is_301(self):
        # Authenticate as user 3
        self.client.login(username='user3', password='donkeykong2018')
        response = self.client.get('/visitorlist/')
        self.assertTrue(response.status_code == 301)
    
    def test_redirect_url_for_user3_on_visitorlist_is_login(self):
        # Authenticate as user 3
        self.client.login(username='user3', password='donkeykong2018')
        response = self.client.get('/visitorlist/', follow=True)
        self.assertRedirects(response, '/login/?next=/visitorlist/', 301)
    
    def test_http_response_is_200_on_user1_get_visitorlist(self):
        self.client.login(username='user1', password='donkeykong2016')
        response = self.client.get('/visitorlist/', follow=True)
        self.assertEqual(response.status_code, 200)

Here we are testing that user3 (which is not assigned as “service account” for any location) will be redirected when visiting the /visitorlist/ url.

We are also testing that the security functionality does not break the user story success for the authorized user, user1, who is assigned as service account for location1.

Here we have checked that the wrong user cannot access the URL without getting redirected, and that it works for the allowed user. If someone changes the logic so that the ownership check is skipped, this test will break. If on the other hand, someone changes the URL configuration so that /visitorlist/ no longer points to this view, the test may or may not break. So being careful about changing the inputs required in tests is important.

Vulnerable open source libraries

According to companies selling scanner solutions for open source libraries, it is one of the most common security problems that people are using vulnerable and outdated libraries. It is definitely easy to get vulnerabilities this way, and as dependencies can be hard to trace manually, having a tool to do so is good. For Python the package safety is a good open source alternative to commercial tools. It is based on the NVD (National Vulnerability Database) from NIST. The database is run by pyup.io and is updated every month (free), or you can pay to get updates faster. If you have a high-stakes app it may pay off to go with a commercial option or to write your own dependency checker.

Running it is as easy as

safety check -r requirements.txt

This will check the dependencies in requirements.txt for known vulnerabilities and give a simple output in the terminal. It can be built into CI/CD pipelines too, as it can export vulnerabilities in multiple formats and also give exit status that can be used in automation.

Static analysis with bandit

Static analysis can check for known anti-patterns in code. A popular choice for looking for vulnerabilities in Python code is bandit. It will test for hardcoded passwords, weak crypto and many other things. It will not catch business logic flaws or architectural bad choices but it is a good help for avoiding pitfalls. Make sure you avoid scanning your virtual environment and tests, unless you want a very long report. Scanning your current project is simple:

bandit -r .

To avoid scanning certain paths, create a .bandit file with defined excludes:

[bandit]
exclude: ./venv/,./*/tests.py

This file will exclude the virtual environment in /venv and all files called “tests.py” in all subfolders of the project directory.

A false positive

Bandit doesn’t know the context of the methods and patterns you use. One of the rules it has is to check if you are using the module random in your code. This is a module is a standard Python modules but it is not cryptographically secure. In other words, creating hashing functions or generating certificates based on random numbers generated by it is a bad idea as crypto analysis could create realistic attacks on the products of such generators. Using the random module for non-security purposes on the other hand, is convenient and unproblematic. Our visitor log app does this, and then bandit tells us we did something naughty:

Test results:
        Issue: [B311:blacklist] Standard pseudo-random generators are not suitable for security/cryptographic purposes.
        Severity: Low   Confidence: High
        Location: ./visitor/views.py:59
        More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b311-random
     58          alco = ''
     59          valcode =  ''.join(random.choice(string.ascii_lowercase) for i in range(6)) 
     60          errmsg = ''    
 

What we see here is the check for cryptographically insecure random numbers. We are using it just to verify that a user knows what they are doing; deleting their own account. The app generates a 6-letter random code that the user has to repeat in text box to delete their account and all associated data. This is not security critical. We can then add a comment # nosec to the line in question, and the scanner will not report on this error again.

Things we will not catch

A static analyser will give you false positives, and there will be dangerous patterns it does not have tests for. There will be things in our threat model we have overlooked, and therefore missing security controls and test requirements. Open source libraries can have vulnerabilities that are not yet in the database used by our scanner tool. Such libraries can also in themselves be malicious by design, or because they have been compromised, and our checks will not catch that. Perhaps fuzzing could, but not always. In spite of this, simple tools like writing a few unit tests, and running some scanners, can remove a lot of weaknesses in an application. Building these into a solid CI/CD pipeline will take you a long way towards “secure by default”.

Commercial VPN’s: the Twitter security awareness flamewar edition

A lot of people worry about information security, and perhaps rightly so. We are steadily plagued by ransomware, data breaches, phishing attacks and password stealers; being reminded of good security habits regularly is generally a good thing. Normally, this does not result on angry people. Except for on the Internet of course, and perhaps in particular on Twitter, the platform made for online rage.

Being angry on the Internet: does a VPN help?

Here’s a recent Tweet from infosec awareness blogger John Opdenakker (you can read his blog here https://johnopdenakker.com):

If you click this one you will get some responses, including some harsh ones:

And another one. Felt like an attack, perhaps it was an attack?

So far the disagreement is not quite clear, just that some people obviously think VPN’s are of little use for privacy and security (and I tend to agree). There are of course nicer ways of stating such opinions. I even tried to meddle, hopefully in a somewhat less tense voice. Or maybe not?

This didn’t really end too well, I guess this was the end of it (not directed at me but at @desdotdev.

This is not a very good way to discuss something. My 2 cents here, beyond “be nice to each other”, was really just a link to this quite good argument why commercial VPN’s are mostly not very useful (except if you want to bypass geoblocking or hide your ip from the websites you visit):

A link to a more sound discussion of the VPN debacle

Risks and VPN marketing

For a good writeup on VPN’s not making you secure, I suggest you read the gist above. Of course, everything depends on everything, and in particular your threat model does. If you fear that evil hackers are sitting on your open WiFi network and looking at all your web traffic to non-https sites, sure, a VPN will protect you. But most sites use HTTPS, and if it is a bank or something similar they will also use HSTS (which makes sure the initial connection is safe too). So what are the typical risks of the coffee shop visiting internet browsing person?

  • Email: malware and phishing emails trying to trick you into sharing too much information or installing malware
  • Magecart infected online shopping venues
  • Shoulder surfers reading your love letters from the chair behind you
  • Someone stealing your phone or laptop while you are trying to fetch that cortado
  • Online bullying threatening your mental health while discussing security awareness on Twitter
  • Secret Chinese agents spying on your dance moves on TikTok

Does a VPN help here? No, it doesn’t. It encrypts the traffic between your computer, and a computer controlled by the VPN company. Such companies typically register in countries with little oversight. Usually the argument is “to avoid having to deliver any data to law enforcement” and besides “we don’t keep logs of anything”. Just completely by coincidence the same countries tend to be tax havens that allows you to hide corporate owner structures as well. Very handy. So, instead of trusting your ISP, you set up a tunnel to a computer entirely controlled by a company owned by someone you don’t know, in a jurisdiction that allows them to do so without much oversight, where they promise not to log anything. I am not sure this is a win for privacy or security. And it doesn’t help against China watching your TikTok videos or a Magecart gang stealing your credit card information on your favourite online store.

One of the more popular VPN providers is ExpressVPN. They provide a 10-step security test, which asks mostly useful questions about security habits (although telling random web pages your preferred messaging app, search engine and browser may not be the best idea) – and it also asks you “do you use a VPN”. If you answer “no” – here’s their security advice for you:

ExpressVPN marketing: do you use a VPN?

It is true that it will make it hard to snoop on you on an open wireless network. But this is not in most people’s threat models – not really. The big problems are usually those in our bullet point list above. ExpressVPN is perhaps one of the least scare-mongering VPN sellers, and even they try to scare you into “but security/privacy anxiety” buying their product. The arguments about getting around geoblocking and hiding your ip from the websites you visit are OK – if you have a need to do that. Most people don’t.

When VPN’s tell you to buy their service to stay safe online, they are addressing a very narrow online risk driver – that is negligible in most people’s threat models.

So what should I do when browsing at a coffee shop?

If you worry about the network itself, a VPN may be a solution to that, provided you trust the VPN itself. You could run your own VPN with a cloud provider if you want to and like to do technical stuff. Or, you could just use your phone to connect to the internet if you have a reasonable data plan. I would rather trust a regulated cell provider than an unregulated anonymous corporation in the Caribbean.

Email, viruses and such: be careful with links and attachments, run endpoint security and keep your computer fully up to date. This takes you a long way, and a VPN does not help at all!

Magecart: this one can be hard to spot, use a credit card when shopping online, and check your statements carefully every month. If your bank provides a virtual card with one-time credit card numbers that is even better. Does a VPN help? No.

Theft of phones, laptops and coffee mugs? Keep an eye on your stuff. Does a VPN help? Nope.

Online bullying? Harder to fight this one but don’t let them get to you. Perhaps John is onto something here? If you feel harassed, use the block button 🙂

Secret Chinese agents on TikTok? No solution there, except not showing your dance moves on TikTok. Don’t overshare. Does a VPN help? Probably not.

Protecting the web with a solid content security policy

We have been used to securing web pages with security headers to fend off cross-site scripting attacks, clickjacking attacks and data theft. Many of these headers are now being deprecated and browser may no longer respect these header settings. Instead, we should be using content security policies to reduce the risk to our web content and its users.

Protect your web resources and your users with Content Security Policy headers!

CSP’s are universally supported, and also allows reporting of policy violations, which can aid in detecting hacking attempts.
Mozilla Developer Network has great documentation on the use of CSP’s: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy.

CSP by example

We want to make it even easier to understand how CSP’s can be used, so we have made some demonstrations for the most common directives we should be using. Let us first start with setting the following header:

Content-Security-Policy: default-src ‘self’;

We have created a simple Flask application to demonstrate this. Here’s the view function:

A simple view function setting a CSP header.

Here we are rendering a template “index.html”, and we have set the default-src directive of the CSP to ‘self’. This is a “fallback” directive in case you do not specify other directives for key resources. Here’s what this does to JavaScript and clickjacking, when other directives are missing:

  • Blocks inline JavaScript (that is, anything inside tags, onclick=… on buttons, etc) and JavaScript coming from other domains.
  • Blocks media resources from other domains, including images
  • Blocks stylesheets from external domains, as well as inline style tags (unless explicitly allowed)

Blocking untrusted scripts: XSS

Of course, you can set the default-src to allow those things, and many sites do, but then the protection provided by the directive will be less secure. A lot of legacy web pages have mixed HTML and Javascript in <script> tags or inline event handlers. Such sites often set default-src: ‘self’ ‘unsafe-inline’; to allow such behaviour, but then it will not help protect against common injection attacks. Consider first the difference between no CSP, and the following CSP:

Content-Security-Policy: default-src: ‘self’;

We have implemented this in a route in our Python web app:

Adding the header will help stop XSS attacks.

Let us first try the following url: /xss/safe/hello: the result is injected into the HTML through the Jinja template. It is using the “safe” filter in the template, so the output is not escaped in any way.

Showing that a URL parameter is reflected on the page. This may be XSS vulnerable (it is).

We see here that the word “hello” is reflected on the page. Trying with a typical cross-site-scripting payload: shows us that this page is vulnerable (which we know since there is no sanitation):

No alert box: the CSP directive blocks it!

We did not get an alert box here, saying “XSS”. The application itself is vulnerable, but the browser stopped the event from happening due to our Content-Security-Policy with the default-src directive set to self, and no script-src directive allowing unsafe inline scripts. Opening the dev tools in Safari shows us a bunch of error messages in the console:

Error messages in the browser console (open dev tools to find this).

The first message shows that the lack of nonce or unsafe-inline blocked execution. This is done by the web browser (Safari).

Further, we see that Safari activates its internal XSS auditor and detects my payload. This is not related to CSP’s, and is internal Safari behavior: it activates its XSS auditor unless there is an X-XSS-Protection header asking to explicitly disable XSS protection. This is Safari-specific and should not be assumed as a default. The X-XSS-Protection header is a security header that has been used in Internet Explorer, Chrome and Safari but it is currently be deprecated. Edge has removed its XSS Auditor, and Firefox has not implemented this header. Use Content Security Policies instead.

What if I need to allow inline scripts?

The correct way to allow inline JavaScript is to include the nonce directive (nonce = number used once) or use a hash of the inline script. These values should then rather be placed in the script-src directive than in the default-src one. For more details on how to do this, see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/script-src#Unsafe_inline_script.

Let’s do an example of an unsafe inline script in our template, using a nonce to allow the inline script. Here’s our code:

Example code showing use of nonce.

Remember to make the nonce unguessable by using a long random number, and make sure to regenerate it each time the CSP is sent to the client – if not, you are not providing much of security protection.

Nonces are only good if they can’t be guessed, and that they are truely used only once.

Here we have one script with a nonce included, and one that does not have it included. The nonce’d script will create an alert box, and the script without the nonce tries to set the inner HTML of the paragraph with id “blocked” to “Hello there”. The alert box will be created but the update of the “blocked” paragraph will be blocked by the CSP.

Here’s the HTML template:

A template with two inline scripts. One with an inserted nonce value, one without. Which one will run?

The result is as expected:

Only the nonce’d script will run 🙂

Conclusion: Use CSP’s for protecting against cross-site scripting (XSS) – but keep sanitising as well: defence in depth.

What about clickjacking?

good explanation of clickjacking and how to defend against it is available from Portswigger: https://portswigger.net/web-security/clickjacking.

Here’s a demo of how clickjacking can work using to “hot” domains of today: who.int and zoom.us (the latter is not vulnerable to clickjacking).

Demo of Clickjacking!

Here’s how to stop that from happening. Add the frame-ancestors directive, and whitelist domains you want to be able of iframing your web page.

Content-Security-Policy: default-src: 'self'; frame-ancestors: 'self' 'youtube.com';

Summary

Protecting against common client-side attacks such as XSS and clickjacking can be done using the Content Security Policy header. This should be part of a defense in depth strategy but it is an effective addition to your security controls. As with all controls that can block content, make sure you test thoroughly before you push it to production!

Is COVID-19 killing your motivation?

The COVID-19 pandemic is taking its toll on all of us. One thing is staying at home, another is the thought of all the things that can go wrong. The virus is very infectious, and is likely to kill a lot of people over the next year. The actions we take, and need to take, to curb the damages of the spreading illness is taking freedoms we take for granted away from us. No more travel, no parties, not even a beer with coworkers. For many of us, even work is gone. No wonder motivation is taking a hit! How can we deal with this situation collectively and individually to make the best out of a difficult situation?

When news are mostly about counting our dead, it can be easy to lose faith in humanity

The virus is not only a risk to our health, it is also a risk to our financial well-being, and the social fabric of our lives. The actions taken to limit the spread of the virus and the load it will have on our healthcare systems, is taking its toll on our social lives, and perhaps also our mental health. It is probably a good idea to think through how important aspects of life will be affected, and what you can do to minimize the risk, and what you should prepare to do if bad consequences do materialize.

TopicRisksThings to do
FinanceJob loss

Real estate value loss
Minimize expenses and build a buffer of money
Ask bank for deferral of principal payments
Plan to negotiate if collateral for mortgage is no longer accepted due to real estate market collapse
Physical healthInfected by COVID-19Supplies in storage at home in case of isolation

Space to isolate to avoid infecting other family members
Mental healthFeeling of isolation

Depression
Avoid “crazy news cycles” and negative feedback on social media

Talk to friends regularly, not just coworkers

Get fresh air and some excercise every day

Have a contact ready for telemedicine, e.g. check if your insurance company offers this
Work Loss of visibility

Degradation of quality

Collaboration problems
Set up daily video calls with closes team members

Make results visible in digital channels

Practice active listening
Example individual risk assessment for COVID-19 life impact

News, social media and fake news

The news cycle is a negative spiral of death counts, stock market crashes and experts preaching the end of the world. While it is useful, and important, to know what the situation is to make reasonable decisions, it is not useful to watch negative news around the clock. It is probably a good idea to batch how much one should take in of the news during a crisis, for example to morning and afternoon news.

Social media tend to paint an even worse picture; taking the news cycle and twisting it into something more extreme. My Twitter feed is now full of people arguing we should go for full communism and introduce death penalties for people allowing children to play outside. It is OK to watch stuff like that a short while for entertainment, but it can easily turn into a force of negative influence and perhaps it would be better to take a break from that? Use filters to stay away from hashtags that bring you down without bringing anything useful.

DevSecOps: Embedded security in agile development

The way we write, deploy and maintain software has changed greatly over the years, from waterfall to agile, from monoliths to microservices, from the basement server room to the cloud. Yet, many organizations haven’t changed their security engineering practices – leading to vulnerabilities, data breaches and lots of unpleasantness. This blog post is a summary of my thoughts on how security should be integrated from user story through coding and testing and up and away into the cyber clouds. I’ve developed my thinking around this as my work in the area has moved from industrial control systems and safety critical software to cloud native applications in the “internet economy”.

What is the source of a vulnerability?

At the outset of this discussion, let’s clarify two common terms, as they are used by me. In very unacademic terms:

  • Vulnerability: a flaw in the way a system is designed and operated, that allows an adversary to perform actions that are not intended to be available by the system owner.
  • A threat: actions performed on an asset in the system by an adversary in order to achieve an outcome that he or she is not supposed to be able to do.

The primary objective of security engineering is to stop adversaries from being able to achieve their evil deeds. Most often, evilness is possible because of system flaws. How these flaws end up in the system, is important to understand when we want to make life harder for the adversary. Vulnerabilities are flaws, but not all flaws are vulnerabilities. Fortunately, quality management helps reduce defects whether they can be exploited by evil hackers or not. Let’s look at three types of vulnerabilities we should work to abolish:

  • Bugs: coding errors, implementation flaws. The design and architecture is sound, but the implementation is not. A typical example of this is a SQL injection vulnerability in a web app.
  • Design flaws: errors in architecture and how the system is planned to work. A flawed plan that is implemented perfectly can be very vulnerable. A typical example of this is a broken authorization scheme.
  • Operational flaws: the system makes it hard for users to do things correctly, making it easier to trick privileged users to perform actions they should not. An example would be a confusing permission system, where an adversary uses social engineering of customer support to gain privilege escalation.

Security touchpoints in a DevOps lifecycle

Traditionally there has been a lot of discussion on a secure development lifecycle. But our concern is removing vulnerabilities from the system as a whole, so we should follow the system from infancy through operations. The following touchpoints do not make up a blueprint, it is an overview of security aspects in different system phases.

  • Dev and test environment:
    • Dev environment helpers
    • Pipeline security automation
    • CI/CD security configuration
    • Metrics and build acceptance
    • Rigor vs agility
  • User roles and stories
    • Rights management
  • Architecture: data flow diagram
    • Threat modeling
    • Mitigation planning
    • Validation requirements
  • Sprint planning
    • User story reviews
    • Threat model refinement
    • Security validation testing
  • Coding
    • Secure coding practices
    • Logging for detection
    • Abuse case injection
  • Pipeline security testing
    • Dependency checks
    • Static analysis
    • Mitigation testing
      • Unit and integration testing
      • Detectability
    • Dynamic analysis
    • Build configuration auditing
  • Security debt management
    • Vulnerability prioritization
    • Workload planning
    • Compatibility blockers
  • Runtime monitoring
    • Feedback from ops
    • Production vulnerability identification
    • Hot fixes are normal
    • Incident response feedback

Dev environment aspects

If an adversary takes control of the development environment, he or she can likely inject malicious code in a project. Securing that environment becomes important. The first principle should be: do not use production data, configurations or servers in development. Make sure those are properly separated.

The developer workstation should also be properly hardened, as should any cloud accounts used during development, such as Github, or a cloud based build pipeline. Two-factor auth, patching, no working on admin accounts, encrypt network traffic.

The CI/CD pipeline should be configured securely. No hard-coded secrets, limit who can access them. Control who can change the build config.

During early phases of a project it is tempting to be relaxed with testing, dependency vulnerabilities and so on. This can quickly turn into technical debt – first in one service, then in many, and at the end there is no way to refinance your security debt at lower interest rates. Technical debt compounds like credit card debt – so manage it carefully from the beginning. To help with this, create acceptable build thresholds, and a policy on lifetime of accepted poor metrics. Take metrics from testing tools and let them guide: complexity, code coverage, number of vulnerabilities with CVSS above X, etc. Don’t select too many KPI’s, but don’t allow the ones you track to slip.

One could argue that strict policies and acceptance criteria will hurt agility and slow a project down. Truth is that lack of rigor will come back to bite us, but at the same time too much will indeed slow us down or even turn our agility into a stale bureaucracy. Finding the right balance is important, and this should be informed by context. A system processing large amounts of sensitive personal information requires more formalism and governance than a system where a breach would have less severe consequences. One size does not fit all.

User roles and stories

Most systems have diffent types of users with different needs – and different access rights. Hackers love developers who don’t plan in terms of user roles and stories – the things each user would need to do with the system, because lack of planning often leads to much more liberal permissions “just in case”. User roles and stories should thus be a primary security tool. Consider a simple app for approval of travel expenses in a company. This app has two primary user types:

  • Travelling salesmen who need reimbursements
  • Bosses who will approve or reject reimbursement claims

In addition to this, someone must be able of adding and removing users, granting access to the right travelling salesmen for a given boss, etc. The system also needs an Administrator, with other words.

Let’s take the travelling salesman and look at “user stories” that this role would generate:

  • I need to enter my expenses into a report
  • I need to attach documentation such as receipts to this report
  • I need to be able of sending the report to the boss for approval
  • I want to see the approval status of my expense report
  • I need to recieve a notification if my report is not approved
  • I need to be able of correcting any mistakes based on the rejection

Based on this, it is clear that the permissions of the “travelling salesman” role only needs to give write access to some operations, for data relating to this specific user, and needs read rights on the status of the approval. This goes directly into our authorization concept for the app, and already here generates testable security annotations:

  • A travelling salesman should not be able to read the expense report of another travelling salesman
  • A travellign salesman should not be able of approving expense reports, including his own

These negative unit tests could already go into the design as “security annotations” for the user stories.

In addition to user stories, we have abusers and abuse stories. This is about the type of adversaries, and what they would like to do, that we don’t want them to be able of achieving. Let’s take as an example a hacker hired by a competitor to perform industrial espionage. We have the adversary role “industrial espionage”. Here are some abuse cases we can define that relate to motivation of a player rather than technical vulnerabilities:

  • I want to access all travel reports to map where the sales personnel of the firm are going to see clients
  • I want to see the financial data approved to gauge the size of their travel budget, which would give me information on the size of their operation
  • I’d like to find names of people from their clients they have taken out to dinner, so we know who they are talking to at potential client companies
  • I’d like to get user names and personal data that allow med to gauge if some of the employees could be recurited as insiders or poached to come work for us instead

How is this hypothetical information useful for someone designing an app to use for expense reporting? By knowing the motivations of the adversaries we can better gauge the credibility that a certain type of vulnerability will be attempted exploited. Remember: Vulnerabilities are not the same as threats – and we have limited resources, so the vulnerabilities that would help attackers achieve their goals are more important to remove than those that cannot easily help the adversary.

Vulnerabilities are not the same as threats – and we have limited resources, so the vulnerabilities that would help attackers achieve their goals are more important to remove than those that cannot easily help the adversary.

Architecture and data flow diagrams

Coming back to the sources of vulnerabilities, we want to avoid vulnerabilities of three kinds; software bugs, software design flaws, and flaws in operating procedures. Bugs are implementation errors, and the way we try to avoid them is by managing competence, workload and stress level, and by use of automated security testing such as static analysis and similar tools. Experience from software reliability engineering shows that about 50% of software flaws are implementation erorrs – the rest would then be design flaws. These are designs and architectures that do not implement the intentions of the designer. Static analysis cannot help us here, because there may be no coding errors such as lack of exception handling or lack of input validation – it is just the concept that is wrong; for example giving a user role too many privileges, or allowing a component to talk to a component it shouldn’t have access to. A good tool for identificaiton of such design flaws is threat modeling based on a data flow diagram. Make a diagram of the software data flow, break it down into components on a reasonable level, and consider how an adversary could attack each component and what could be the impact of this. By going through an excercise like this, you will likely identify potential vulnerabilities and weaknesses that you need to handle. The mitigations you introduce may be various security controls – such as blocking internet access for a server that only needs to be available on the internal network. The next question then is – how do you validate that your controls work? Do you order a penetration test form a consulting company? That could work, but it doesn’t scale very well, you want this to work in your pipeline. The primary tools to turn to is unit and integration testing.

We will not discuss the techniques for threat modeling in this post, but there are different techniques that can be applied. Keep it practical, don’t dive too deep into the details – it is better to start with a higher level view on things, and rather refine it as the design is matured. Here are some methods that can be applied in software threat modeling:

Often a STRIDE-like approach is a good start, and for the worst case scenarios it can be worthwhile diving into more detail with attack trees. An attack tree is a fault tree applied to adversarial modeling.

After the key threats have been identified, it is time to plan how to deal with that risk. We should apply the defense-in-depth principle, and remeber that a single security control is usually not enough to stop all attacks – because we do not know what all possible attack patterns are. When we have come up with mitigations for the threats we worry about, we need to validate that they actually work. This validation should happen at the lowest possible level – unit tests, integration tests. It is a good idea for the developer to run his or her own tests, but these validations definitely must live in the build pipeline.

Let’s consider a two-factor authentication flow using SMS-based two-factor authentication. This is the authentication for an application used by politicians, and there are skilled threat actors who would like to gain access to individual accounts.

A simple data flow diagram for a 2FA flow

Here’s how the authentication process work:

  • User connects to the domain and gets an single-page application loaded in the browser with a login form with username and password
  • The user enters credentials, that are sent as a post request to the API server, which validates it with stored credentials (hashed in a safe way) in a database. The API server only accepts requests from the right domain, and the DB server is not internet accessible.
  • When the correct credentials have been added, the SPA updates with a 2fa challenge, and the API server sends a post request to a third-party SMS gateway, which sends the token to the user’s cell phone.
  • The user enters the code, and if valid, is authenticated. A JWT is returned to the browser and stored in localstorage.

Let’s put on the dark hat and consider how we can take over this process.

  1. SIM card swapping combined wiht a phishing email to capture the credentials
  2. SIM card swapping combined with keylogger malware for password capture
  3. Phishing capturing both password and the second factor from a spoofed login page, and reusing credentials immediately
  4. Create an evil browser extension and trick the user to install it using social engineering. Use the browser extension to steal the token.
  5. Compromise a dependency used by the application’s frontend, to allow man-in-the-browser attacks that can steal the JWT after login.
  6. Compromise a dependency used in the API to give direct access to the API server and the database
  7. Compromise the 3rd party SMS gateway to capture credentials, use password captured with phishing or some other technique
  8. Exploit a vulnerability in the API to bypass authentication, either in a dependency or in the code itself.

As we see, the threat is the adversary getting access to a user account. There are many attack patterns that could be used, and only one of them involves only the code written in the application. If we are going to start planning mitigations here, we could first get rid of the two first problems by not using SMS for two-factor authenticaiton but rather relying on an authenticator app, like Google Authenticator. Test: no requests to the SMS gateway.

Phishing: avoid direct post requests from a phishing domain to the API server by only allowing CORS requests from our own domain. Send a verification email when a login is detected from an unknown machine. Tests: check that CORS from other domains fail, and check that an email is sent when a new login occurs.

Browser extensions: capture metadata/fingerprint data and detect token reuse across multiple machines. Test: same token in different browsers/machines should lead to detection and logout.

Compromised dependencies is a particularly difficult attack vector to deal with as the vulnerability is typically unknown. This is in practice a zero-day. For token theft, the mitigation of using meta-data is valid. In addition it is good practice to have a process for acceptance of third-party libraries beyond checking for “known vulnerabilities”. Compromise of the third-party SMS gateway is also difficult to deal with in the software project, but should be part of a supply chain risk management program – but this problem is solved by removing the third-party.

Exploit a vulnerability in the app’s API: perform static analysis and dependency analysis to minimize known vulnerabilities. Test: no high-risk vulnerabilities detected with static analysis or dependency checks.

We see that in spite of having many risk reduction controls in place, we do not cover everything that we know, and there are guaranteed to be attack vectors in use that we do not know about.

Sprint planning – keeping the threat model alive

Sometimes “secure development” methodologies receive criticims for “being slow”. Too much analysis, the sprint stops, productivity drops. This is obviously not good, so the question is rather “how can we make security a natural part of the sprint”? One answer to that, at least a partial one, is to have a threat model based on the overall architecture. When it is time for sprint planning, there are three essential pieces that should be revisited:

  • The user stories or story points we are addressing; do they introduce threats or points of attack not already accounted for?
  • Is the threat model we created still representative for what we are planning to implement? Take a look at the data flow diagram and see if anything has changed – if it has, evaluate if the threat model needs to be updated too.
  • Finally: for the threats relevant to the issues in the sprint backlog, do we have validation for the planned security controls?

Simply discussing these three issues would often be enough to see if there are more “known uknowns” that we need to take care of, and will allow us to update the backlog and test plan with the appropriate annotations and issues.

Coding: the mother of bugs after the design flaws have been agreed upon

The threat modeling as discussed above has as its main purpose to uncover “design flaws”. While writing code, it is perfectly possible to implement a flawed plan in a flawless manner. That is why we should really invest a lot of effort in creating a plan that makes sense. The other half of vulnerabilities are bugs – coding errors. As long as people are still writing code, and not some very smart AI, errors in code will be related to human factors – or human error, as it is popularly called. This often points the finger of blame at a single individual (the developer), but since none of us are working in vacuum, there are many factors that influence these bugs. Let us try to classify these errors (leaning heavily on human factors research) – broadly there are 3 classes of human error:

  • Slips: errors made due to lack of attention, a mishap. Think of this like a typo; you know how to spell a word but you make a small mistake, perhaps because your mind is elsewhere or because the keyboard you are typing on is unfamiliar.
  • Competence gaps: you don’t really know how to do the thing you are trying to do, and this lack of knowledge and practice leads you to make the wrong choice. Think of an inexperienced vehicle driver on a slippery road in the dark of the night.
  • Malicious error injection: an insider writes bad code on purpose to hurt the company – for example because he or she is being blackmailed.

Let’s leave the evil programmer aside and focus on how to minimize bugs that are created due to other factors. Starting with “slips” – which factors would influence us to make such errors? Here are some:

  • Not enough practice to make the action to take “natural”
  • High levels of stress
  • Lack of sleep
  • Task overload: too many things going on at once
  • Outside disturbances (noise, people talking to you about other things)

It is not obvious that the typical open office plan favored by IT firms is the optimal layout for programmers. Workload management, work-life balance and physical working environment are important factors for avoiding such “random bugs” – and therefore also important for the security of your software.

These are mostly “trying to do the right thing but doing it wrong” type of errors. Let’s now turn to the lack of competence side of the equation. Developers have often been trained in complex problem solving – but not necessarily in protecting software from abuse. Secure coding practices, such as how to avoid SQL injection, why you need output escaping and similar types of practical application secuity knowledge, is often not gained by studying computer science. It is also likely that a more self-taught individual would have skipped over such challenges, as the natural focus is on “solving the problem at hand”. This is why a secure coding practice must deliberately be created within an organization, and training and resources provided to teams to make it work. A good baseline should include:

  • How to protect aginst OWASP Top 10 type vulnerabilities
  • Secrets management: how to protect secrets in development and production
  • Detectability of cyber threats: application logging practices

An organization with a plan for this and appropriate training to make sure everyone’s on the same page, will stand a much better chance of avoiding the “competence gap” type errors.

Security testing in the build pipeline

OK, so you have planned your software, created a threat model, commited code. The CI/CD build pipeline triggers. What’s there to stop bad code from reaching your production environment? Let’s consider the potential locations of exploitable bugs in our product:

  • My code
  • The libraries used in that code
  • The environment where my software runs (typically a container in today’s world)

Obviously, if we are trying to push something with known critical errors in either of those locations to production, our pipeline should not accept that. Starting with our own code, a standard test that can uncover many bugs is “static analysis”. Depending on the rules you use, this can be a very good security control but it has limitations. Typically it will find a hardcoded password written as

var password = 'very_secret_password";

but it may not find this password if it isn’t a little bit smart:

var tempstring = 'something_that_may_be_just_a_string";

and yet it may throw an alert on

var password = getsecret();

just because the word “password” is in there. So using the right rules, and tuning them, is important to make this work. Static analysis should be a minimum test to always include.

The next part is our dependencies. Using libraries with known vulnerabilities is a common problem that makes life easy for the adversary. This is why you should always scan the code for external libraries and check if there are known vulnerabilitie sin them. Commercial vendors of such tools often refer to it as “software component analysis”. The primary function is to list all dependencies, check them against databases of known vulnerabilities, and create alerts accordingly. And break the build process based on threshold limits.

Also the enviornment we run on should be secure. When building a container image, make sure it does not contain known vulnerabilities. Using a scanner tool for this is also a good idea.

While static analysis is primarily a build step, testing for known vulnerabilities whether in code libraries or in the environment, should be done regulary to avoid vulnerabilities discovered after the code is deployed from remaining in production over time. Testing the inventory of dependencies against a database of known vulnerabiltiies regulary would be an effective control for this type of risk.

If a library or a dependency in the environment has been injected with malicious code in the supply chain, a simple scan will not identify it. Supply chain risk management is required to keep this type of threat under control, and there are no known trustworthy methods of automatically identifying maliciously injected code in third-party dependencies in the pipeline. One principle that should be followed with respect to this type of threat, however, is minimization of the attack surface. Avoid very deep dependency trees – like an NPM project 25000 dependencies made by 21000 different contributors. Trusting 21000 strangers in your project can be a hard sell.

Another test that should preferably be part of the pipeline, is dynamic testing where actual payloads are tested against injection points. This will typically uncover other vulnerabilities than static analysis will and is thus a good addition. Note that active scanning can take down infrastructure or cause unforeseen errors, so it is a good idea to test against a staging/test environment, and not against production infrastructure.

Finally – we have the tests that will validate the mitigations identified during threat modeling. Unit tests and integration tests for securtiy controls should be added to the pipeline.

Modern environments are usually defined in YAML files (or other types of config files), not by technicians drawing cables. The benefit of this, is that the configuration can be easily tested. It is therefore a good idea to create acceptance tests for your Dockerfiles, Helm charts and other configuration files, to avoid an insider from altering it, or by mistake setting things up to be vulnerable.

Security debt has a high interest rate

Technical debt is a curious beast: if you fail to address it it will compound and likely ruin your project. The worst kind is security debt: whereas not fixing performance issues, removing dead code and so on compunds like a credit card from your bank, leaving vulnerabilities in the code compunds like interest on money you lent from Raymond Reddington. Manage your debt, or you will go out of business based on a ransomware compaign followed by a GDPR fine and some interesting media coverage…

You need to plan for time to pay off your technical debt, in particular your securiyt debt.

Say you want to plan using a certain percentage of your time in a sprint on fixing technical debt, how do you choose which issues to take? I suggest you create a simple prioritization system:

  • Exposed before internal
  • Easy to exploit before hard
  • High impact before low impact

But no matter what method you use to prioritize, the most important thing is that you work on getting rid of known vulnerbilities as part of “business-as-usual”. To avoid going bankrupt due to overwhelming technical debt. Or being hacked.

Sometimes the action you need to take to get rid of a security hole can create other problems. Like installing an update that is not compatible with your code. When this is the case, you may need to spend more resources on it than a “normal” vulnerability because you need to do code rewrites – and that refactoring may also need you to update your threat model and risk mitigations.

Operations: your code on the battle field

In production your code is exposed to its users, and in part it may also be exposed to the internet as a whole. Dealing with feedback from this jungle should be seen as a key part of your vulnerability management program.

First of all, you will get access to logs and feedback from operations, whether it is performance related, bug detections or security incidents. It is important that you feed this into your issue management system and deal with it throughout sprints. Sometimes you may even have a critical situation requiring you to push a “hotfix” – a change to the code as fast as possible. The good thing about a good pipeline is that your hotfix will still go through basic security testing. Hopefully, your agile security process and your CI/CD pipeline is now working so well in symbiosis that it doesn’t slow your hotfix down. In other words: the “hotfix” you are pushing is just a code commit like all others – you are pushing to production several times a day, so how would this be any different?

Another aspect is feedback from incident response. There are two levels of incident response feedback that we should consider:

  1. Incident containment/eradication leading to hotfixes.
  2. Security improvements from the lessons learned stage of incident response

The first part we have already considered. The second part could be improvements to detections, better logging, etc. These should go into the product backlog and be handled during the normal sprints. Don’t let lessons learned end up as a PowerPoint given to a manager – a real lesson learned ends up as a change in your code, your environment, your documentation, or in the incident response procedures themselves.

Key takeaways

This was a long post, here are the key practices to take away from it!

  • Remember that vulnerabilities come from poor operational practices, flaws in design/architecture, and from bugs (implementation errors). Linting only helps with bugs.
  • Use threat modeling to identity operational and design weaknesses
  • All errors are human errors. A good working environment helps reduce vulnerabilities (see performance shaping factors).
  • Validate mitigations using unit tests and integration tests.
  • Test your code in your pipeline.
  • Pay off technical debt religiously.