Deploying Django to app engine with Python 3.7 runtime – fails because it can’t find pip?

Update 1 June 2020: Google tells me the problem is now fixed. I haven’t verified it yet, will update if I find it is not fixed at the next crossroads. Bug tracker link:

Update 30 March 2020: The problem is still here. Use pip version 19.2.3 now to make it work. Google: can you please go fix this now?

Update 30 April 2019: Problem is back. This time it tries to upgrade to pip 19.1, but the app engine instance is stuck on 19.0.3. Adding pip==19.0.3 in the requirements.txt file saves the deployment.

Update 1 April 2019: now the deploy fails with the same message as described in this post, when the PIP version is specified in the requirements.txt file. Removing the specific pip version line from the requirements file fixes this. I have not seen any change notice or similar from Google on this.

I had an interesting error that took quite some time to hunt down today. These are basically some notes on what caused it and how I tracked it down. I have an app that is deployed to Google App Engine standard. It is running Django and using the Python 3.7 runtime – and it was working quite well for some time. Then yesterday I was going to deploy an update (actually just adding some CSS tweaks), it failed, with a cryptic error message. Running “gcloud app deploy” lead to the following error message:

File upload done.
Updating service [default]…failed.
ERROR: ( Error Response: [9] Cloud build a33ff087-0f47-4f18-8654-********* status: FAILURE.
Error ID: B212CE0B.
Error type: InternalError.
Error message: pip_install_from_wheels had stderr output:
/env/bin/python3.7: No module named pip
error: pip_install_from_wheels returned code: 1.

This is weird: this is a normal Python project using a requirements.txt file for its dependencies. The file was generated using pip freeze, and should not contain anything weird (it doesn’t). Searching the wisdom of the internet reveals that nobody else seems to have this problem, and it only occurred since yesterday. My hunch was that they’ve changed something on the GAE environment that broke something. Searching the net gives us these options:

  • The requirements.txt file has weird encoding and contains Chinese signs/letters? That was not it.
  • This is because you need to install some special packages for using Python3.. was also not the case and would have been weird changing since a few days ago…
  • You need to manually install pip to make it work – which may be the case sometimes but without SSH access to the instance this isn’t obvious how to do.
The trick is often looking at the right logs….

So, being clueless I turned to looking for the right logs to figure out what is going on. Not being an expert on the GAE environment led to some hunting in the web console until I found “Cloud build”, which sounded promising. That was the right place to be – GAE uses a build process in the cloud to first build the application, and then a Docker image to push to the internal Google Cloud Docker repository. Hunting the build log for this finds this little piece of gold:

Step #1 - "builder": INFO pip_install_from_wheels took 0 seconds
Step #1 - "builder": INFO starting: pip_install_from_wheels
Step #1 - "builder": INFO pip_install_from_wheels /env/bin/python3.7 -m pip install --no-deps --prefix /tmp/tmp9Y3aD7/env /tmp/tmppuvw4s/wheel/pip-19.0.1-py2.py3-none-any.whl --disable-pip-version-check
Step #1 - "builder": INFO `pip_install_from_wheels` stdout:
Step #1 - "builder": Processing /tmp/tmppuvw4s/wheel/pip-19.0.1-py2.py3-none-any.whl
Step #1 - "builder": Installing collected packages: pip
Step #1 - "builder": Found existing installation: pip 18.1
Step #1 - "builder": Uninstalling pip-18.1:
Step #1 - "builder": Successfully uninstalled pip-18.1
Step #1 - "builder": Successfully installed pip-19.0.1
Step #1 - "builder": Failed. No module /env/python/pip

Before the weird error we see it is trying to uninstall pip-18.1, then install pip-19.0.1 (a more recent version), and then it can’t find pip afterwards and the build process fails. This has not been configured by me and is probably Google configuring automatic upgrades of some packages during build – and here it breaks the workflow.

Fixing it

The temporary fix was simple. Adding “pip==18.1” to the requirements.txt file, allowed the build process to run and it deployed nicely.

What did we learn from this?

  • API tools give only partial error messages, making debugging hard.
  • Automated upgrade configs are good but can cause things to break in unforeseen ways.
  • Finding the right logs is the key to fixing weird problems

Combining VueJS and Django to build forms with custom widgets

This post is brief and explains a pattern that may be dangerous but still is very handy for combining VueJS with Django templates for dynamic forms. Here’s the case: I need to build a form for sending out some messages. One of the form widgets is a <select> tag where each <option> is a model instance from Django. The widget will then show the name of that model instance in the UI, but this does not provide enough context to be useful, we also need some description text. There are basically two options for how to handle this:

  1. Use the form “as-is” but provide the extra context in the UI by pulling some extra information and building an information box in the UI.
  2. Create a custom widget, and bind it to the Django model form using a hidden field.

Both are probably equally good, but I went with the second option. So here’s what I did:

  1. Build a normal Django model form, but change the widget for the field in question to type “HiddenInput” in the file.
  2. Build a selector widget using VueJS that allows the user to get the desired content and review the various options with full context (including images and videos, things you can’t put inside a dropdown list. We are binding the selected choices to frontend data using the v-model directive in VueJS.
  3. Set the hidden field to set its value based on the data value stored in the frontend using that v-model directive
  4. Process the form as you normally would with a Django model form.

The form definition remains very simple. Here’s the relevant class from this example:

class MailForm(forms.ModelForm):

    class Meta:
        model = Campaign
        fields = ('name','to','elearning',)
        widgets = {
            'elearning': forms.HiddenInput(attrs={':value': ''})

The selector widget can take any form you could desire. The point in this project was to show some more context for the “eLearning” model. The user here gets notification about enrollment in an eLearning module by e-mail. The administrator setting up the program needs to get a bit of context about that e-learning, such as the name of the module, a description of its content, and perhaps a preview of a video or other multimedia. Below is an example of a simple widget of this type. The course administrator can here browse through the various options by clicking next, and the e-mail form is automatically updated.

Of course, to do that binding we need a bit of JavaScript in the Django template. We need to perform the following tasks to make our custom widget work:

  1. Fetch information about all the options from the server. We need to create an API endpoint for this, that can deliver JSON data to the frontend.
  2. Set the data item bound to the Django form based on the user’s current selection

Now the form can be submitted and processed using the normal Django framework patterns – but with a much more context-rich selection widget than a simple dropdown list.

Is it safe to do this?

Combining frontend and server-side rendering with different templates for HTML rendering can be dangerous. See this excellent write-up on XSS vulnerabilities that can be the result from such combinations:

This is a problem when user input is injected via the server-side template as the user can supply the interpolation tags as part of the input. In our case there is no user input in those combinations. However, if you need to take user input and rerender this using the server-side templates of some framework like Django, here are some things you can do to harden against this threat:

  • Use the v-pre directive in VueJS
  • Sanitize the input to discard unsafe characters, including VueJS delimiters
  • Escape generated output from the database to avoid injections making it as executable JavaScript reaching the user’s context

Security awareness: the tale of the Minister of Fisheries and his love of an Iranian former beauty queen

An interesting story worthy of inspiring books and TV shows is unfolding in Norway. The Minister of Fisheries, Per Sandberg (born 1960), from the Progress Party (a populist right party), spent his summer holiday in Iran together with his new girlfriend, a 28-year old former beauty queen who fled to Norway do escape forced marriage when she was 16. The minister brought his smartphone, where he has access to classified information systems. He forgot to inform the prime minister before after he left, a breach of security protocol. He ignored security advice from the Norwegian security police, responsible for national security issues and counter-intelligence. He is still a member of the cabinet. This post is an attempt at making sense of this, and what the actual risk is. A lot of people in Norway have had their say in media about this case, both knowledgeable voices, and less reasonable ones.

Some context: Norwegian-Iranian relations

Traditionally there has been little trade between Iran and Norway. Recently, following the nuclear agreement between Iran and the US, UK, France, China, Russia and Germany this has started to change. Norway has seen significant potential for exports to Iran of fish and aquaculture technologies. In the last year or so, Minister Sandberg has been central to this development (see timeline further down on Sandberg’s known touch points with Iran).

In the Norwegian public skepticism of the Iranian regime is high, and there has been vocal criticism of establishing trade relationships over human rights concern.

The Norwegian Iranian interest spheres are also intersecting in the Middle East. Iran has established tighter relations with Russia since 2016 when it started to allow Russian bombers to take off from Iranian air force bases for bombing missions inside Syria. The Norwegian-Russian relations are strained, following the response of NATO and the EU to Russian operations in the Ukraine, influencing of western elections and a general intensification of cyber operations against Norwegian targets (see open threat assessment from Norwegian military intelligence: (in Norw.). Operations against Norwegian government officials by Iranian services may thus also be driven by other interests of Iran than direct Norwegian-Iranian relations.

Sandberg: who is he and what could make him a viable target for intelligence operations?

This is a presentation of Sandberg taken from the web page of the Ministry of Trade, Industry and Fisheries ( Note that his marital status is “married” – but he separated from his wife in May 2018. Sandberg is a vocal figure in Norwegian politics. He has been known to be against immigration and a supporter of strict immigration laws. He has repeatedly been accused of racism, especially by the opposition. He has long held top positions in the Progress Party, which has been a part of a coalition cabinet together with the conservatives (Høyre), and more recently also with the moderately liberalist party “Venstre” (meaning left but it is not a socialist party). Sandberg is known for multiple controversies, summarized on this Wikipedia page: This involves addressing the parliament after having too much to drink, losing his driver’s license due to speeding and finally he was also convicted for violence against an asylum seeker in 1997.

Sandberg has been married since 2010 to 2018 to Line Miriam Sandberg, who has been working as a state secretary for the Ministry of Health since 2017. They recently separated.

His new girlfriend

Sandbergs new girlfriend came to Norway when she was 16 (or 13/14 the first time according to some sources) to flee forced marriage to a 60-year old man in Iran. She is now a Norwegian citizen and is 28 years old. She has participated in several beauty contests in 2013-2014. After she first came to Norway, she was not granted asylum and returned to Iran. Iran sent her back to Norway again because she did not have any identification papers when arriving, and she was adopted by a Norwegian family. A summary of known facts about Letnes and how she gained access to Iran after being returned to Norway without ID papers when she was a teenager was written in Norwegian by Mahmod Fahramand ( Farahmand is currently a consultant with the auditing and consulting firm BDO and has background from the Norwegian armed forces. He often writes opinion pieces about security related topics. To summarize some of Farahmand’s points.

  • Letnes was returned to Norway and was later adopted by her foster family
  • She has been a “go-to-person” for journalists wanting to get in touch with Iranian officials and has been known to have close relationships with the Iranian embassy in Oslo
  • Iran does not allow Iranian-born indivdiuals to enter Iran without an Iranian passport. If they do not have this, they will need to get access to their birth certificate or otherwise prove to the Iranian government that they in fact have a right to an Iranian passport. Since Letnes fled Iran to seek protection from the threat of her family, it seems she must have gotten access to this without contacting her family, Farahmand argues.

Letnes had her application for asylum turned down 3 times before getting it approve. The reason for the change of the decision of the immigration authorities in 2008 is not known (Norw:–jeg-er-kjempeglad-1.6236578). In addition, it has become known in media in the last few days that Letnes applied for a job with Sandberg’s ministry, suggesting she could act as a translator and guide for Sandberg’s communications with Iran in matters related to fishery and aquaculture trade, which she did not get. Sandberg denied any knowledge of this prior to media inquiring about it. The job application was sent in 2016. She also registered a sole proprietorship in January this year, B & H GENERAL TRADING COMPANY. BAHAREH LETNES, a company to trade with Iran in fish, natural gas, oil and technology (corporate registration information: The company has according to Letnes not had any activity so far, according to media reports.

A honeytrap? Possibly. A security breach? For sure.

The arguments from Farahmand’s article above, together with the fact that Letnes tried to get a job for Sandberg in 2016, could easily indicate that Letnes sought to get close to Sandberg. She has sought multiple touchpoints with him since he was appointed Minister of Fisheries in 2015.

This would be a classical honeytrap, although a relatively public one. Sandberg has failed to follow security protocol on many occasions in his dealings with Letnes and Iran. Obvious signs of poor security awareness on behalf of the Minister:

  • He brought his government issued cell phone to Iran and left it unattended for longer periods of time where they stayed at the time
  • He did not tell the office of the Prime Minister about his travel to Iran before leaving. This is a breach of security protocol for Norwegian ministers
  • His separation from his wife became known in May this year
  • He has announced his “original vacation plans got smashed, so the trip to Iran was a last-minute decision”. He was supposed to go on holiday to Turkey, which he had also reported to his Ministry and the office of the Prime Minister, in accordance with security protocol (Norw: )
  • The Norwegian government was made aware of Sandberg’s presence in Iran when they received an e-mail from the Iranian embassy in Oslo, requesting official meetings with Minister Sandberg while he was in Iran

Iranian TTP

According to Kjell Grandhagen, former head of Norwegian military intelligence, Iran has a very capable and modern intelligence organization. He holds it as highly likely that Sandberg’s government issued phone, which he left unattended a lot of the time while in Iran, has been hacked ( According to this CSO summary, Iran has serious capabilities within both HUMINT and cyber domains. Considering the known cyber capabilities of Iran, and the looming sanctions from the Trump administration, getting both information and leverage over a key politician in a NATO country becomes even more interesting, not only to Iran but also to Russia.

Coming back to Iran’s more recent tighter cooperation with Russia, it is not unlikely that they are also initiating a closer relationship when it comes to intelligence gathering. The use of honey traps has been a long-standing Russian tactic for information gathering and getting leverage over decision makers. In 2015, Norwegian police warned against Russian intelligence operations targeting politicians, including the use of honey traps (

A summary: why is he still in office?

The facts and arguments presented above should indicate two things very clearly:

  • Based on publicly known information, it is clearly possible that Iranian intelligence is targeting Per Sandberg. They may have an asset close to him, as well as having had physical access to his smartphone that has direct access to classified information systems.
  • Further, Sandberg has broken established security protocol, and although admitting this, he does not seem to appreciate the potential impact

The effect of a top leader not taking security seriously is very unfortunate. Good security awareness in an organization depends heavily on the visible actions of its people at the top – in business as well as in politics. A breach of security policy without getting any personal consequences on this level sends a very poor message to other politicians and government officials. It also sends a message to adversaries that targeting top-level politicians is likely to work, even if there are numerous indicators of a security breach. There should be no other possible conclusion of this than relieving Mr. Sandberg of his position – which would set him free to further develop his relationship with the Iranian beauty queen.

Making Django, Elastic Beanstalk and AWS RDS play well together

A couple of days ago I decided I should learn a bit more hands-on AWS stuff. So I created a free tier AWS account, and looked around. I decided I’d take a common use case; deploy a web application to Elastic Beanstalk and add a domain and SSL.

Setting up tools

Step 1: reading documentation. AWS has a lot of documentation, and it is mostly written in a friendly manner with easy-to-follow instructions. Based on the documentation I opted for using the command line Elastic Beanstalk tool. To use this you need Python and pip. You can install it with the command

pip install awsebcli –upgrade

If you are having a permissions problem with doing this, you can throw in a “–user” flag at the end of that command. This will install the tool you need to create and manage EB environments from your command line. Since it is a Python utility it works on Windows, as well as Mac and Linux. Installing this did not pose any hiccups. You can read more about how to set this tool up and updating your system path here:

Before using it you need to set it up. Issue the command

eb init.

This will give you a prompt asking for a number of things, like region to set up in, etc.

Learning point: if you want to set up a database, such as MySQL in the EB environment, you should use the database option when issuing the next command. Anyway to set up your environment, use

eb create

If you want a database in your environment add the –db flag with the desired options; you cannot create the database in the EB Console (web-based interface) afterwards, at least not for micro instances that are allowed in the free tier. According to someone on Stack Overflow, this is a bug in AWS that you can wait for them to fix – or use the command line option (supposedly that works but it is not what I did).

If you create a database in your EB environment, your DB will be terminated too if you terminate that environment. You may not want that, so you can consider setting up an external database and connecting to it outside of EB. That is what I did, and there’s more about that a little further down this post.

Creating a Django app

To have something to deploy I created a Django app. This is an asocial network; you can post things with links and hashtags but you can’t follow other users or anything like that. It has user management and uses the default Django admin system and authentication system (session based). I called it woodscreaming and you can view it here:

Setting up a virtual environment

First, to avoid mixing up things and creating a requirements file that works, create a virtual environment. For this I like to use the tool virtualenv (works on all platforms, can be installed with pip if you don’t have it):

virtualenv –python=python venv

“venv” is the name of your virtual environment. Everything you install when the environment is active will be contained in that environment, and you have all dependencies under control (think of it like a semi-container-solution). To activate the environment on Linux/Mac:

source venv/bin/activate

On Windows:


When you have all the dependencies your app needs in place, run

pip freeze > requrements.txt

This creates a requirements.txt file that EB will use to install your app in the cloud.

Adding EB configuration files to the Django project

To make things work, you also need to add some EB specific configuration files to your Django project. Create a folder named .ebextensions in your project’s root folder. In this folder you will need to add a django.config file with the following contents:

    WSGIPath: projectname/

Of course you need to change the word projectname into the name of your project. This tells EB where to find your wsgi file. This file describes how the web server should be set up and is a Python standard.

You should also tell EB to run migrations to get your data models to work with your database. Adding a file (I called it db-migrate.config) to the .ebextensions folder fixes this. Here’s what you need to add to that file:

    command: " migrate"
    leader_only: true
    DJANGO_SETTINGS_MODULE: discproject.settings

You should also create a folder called .elasticbeanstalk. The command line client will populate this with a YAML file called config.yml tha tells EB what resources are needed (you don’t need to edit this file yourself).

That’s it to begin with – some changes need to be made when adding an RDS database and setting up http to https forwarding.

Deploying to EB

Deploying to EB is very easy, you simply deactivate your virtual environment by issuing the command “deactivate” and then you run

eb deploy

It now zips your source, uploads it to AWS and installs it and provisions the resources defined in your config.yml file. It takes a while, and then it is done. Then you can see your web app online by issuing the command

eb open

The app will get its own URL automatically, of the format “”. It does not get an SSL certificate (https) automatically – you will need to set up a custom domain for that (more about that later). Anyway, opening it up shows the web app running in the cloud and I am able to use it.

Dev database vs prod database

By default sets up a project hat uses an SQLite database; a single file SQL database that is popular for persistent storage in mobile apps and embedded applications. When deploying your development environment’s database is deployed too, and with each redeploy you will overwrite it. It is not great for concurrent operations, and obviously overwriting all user data on each deploy is not going to work. There are ways around this if you want to stick to SQLite but that is normally not the best solution for a web app database (although it is great for development).

Next we look at how we can create a database in the cloud and use that with our production environment, while using the SQLite one in local development.

Adding an RDS database

Attempt 1: Using the EB Console

In the EB console (the web interface), if you go to “Configuration”, there is a card for “Database” and an option to “modify”. There you can set up your desired database instance and select apply. The problem is… it doesn’t work for some reason. The deployment fails due to some permission error. I’m sure it is possible to fix but I didn’t bother fiddling enough with it to do that. And as mentioned above; if you terminate the environment you will also terminate the database.

Attempt 2: Setting up and RDS database external to EB

This worked. Basically following AWS documentation on how to set it up was quick and easy:

  • Go to RDS, create a new instance. Select the type of database engine, EC2 instance type etc.
  • Select db name, username, password (remember to write those down – I use secure notes in LastPass for things like this). Set the DB instance to be “public” to allow queries from outside your VPC to reach it.
  • Add the RDS security group to your EB EC2 instance. This is important – if you do not do this, it is not possible to query the database from EB.

To add that security group in EB you need to go to the EB console, head to configuration and then select the card for instances. Select “modify” and then head to the security groups table – add the RDS one (it is automatically generated and named something like rds-default-1) and click apply. Because the database is external to EB you also need to add environment variables for the connection. To do this, head to the console again and select “modify” on the software card. Add the following environment variables:


The values are found in your RDS instance overview (head to the RDS console, select your instance, and you find the variables a bit down on the page). Now, you also need to tell your Python app to read and use these. Add this to your Django settings file:

if 'RDS_HOSTNAME' in os.environ:
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'NAME': os.environ['RDS_DB_NAME'],
            'USER': os.environ['RDS_USERNAME'],
            'PASSWORD': os.environ['RDS_PASSWORD'],
            'HOST': os.environ['RDS_HOSTNAME'],
            'PORT': os.environ['RDS_PORT'],
        'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'dbp.sqlite3'),

After doing this the EB environment health was showing as “green” and all good. But my web app did not show up and the log showed database connection errors. The solution to that was: read the docs. You also need to add the RDS default security group (the one that allows inbound connections) to the allowed sources for inbound connections. Details here: After doing this – it works!

Adding a Django superuser to the RDS database

You could SSH into your EC2 instance running the Django app and use the utility; but this kind of beats the point of having a PaaS that supposedly should be able to configure things without SSH-ing into everything.

To add a Django superuser you should thus add a new Django command to your environment. Here’s a good description of how to do that: You can add the command to your db-migrate.config file in the .ebextensions folder.

Configuring DNS with Route 53

Now, having the default URL is no fun, and you can’t add SSL on that one. So we need to set up DNS. I chose to buy a domain name from Amazon and then set up DNS with Route 53. Setting that up for an EB environment is super-easy: you make an A record as alias to your EB environment URL.

Adding an SSL certificate that terminates on the load balancer

Now that we have a working domain name, and we’ve set up the DNS records we need, we can add an SSL certificate. The easiest way to provision the certificate is to use Amazons certificate management service. You provision one for your domain, and you can verify by adding a CNAME record to your DNS hosted zone in Route 53.

The next thing you need to do to make things work is add the certificate to your Elastic Beanstalk environment. Depending on your threat model and your needs, you can choose the simple route of terminating https on the load balancer (good enough for most cases), or you can set up AWS to also use secure protocols in internal traffic (behind the load balancer). I chose to terminate traffic on the load balancer.

The AWS docs explains how to do this by adding a secure listener on the load balancer:

Forwarding http to https

To forward http traffic to https there are several ways this can be done. The easiest is to set up forwarding on the Apache server. Since we are not using SSH to fiddle with the server directly, we do this by adding a configuration file to our .ebextensions folder in the Django project, and then redeploying. Adding a file https.config with the following contents does the job:

        mode: "000644"
        owner: root
        group: root
        content: |
            RewriteEngine On
            <If "-n '%{HTTP:X-Forwarded-Proto}' && %{HTTP:X-Forwarded-Proto} != 'https'">
            RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R,L]


This post is a walk-through of getting the essentials done to use Elastic Beanstalk to serve a web application:

  • Create an environment and deploy an app
  • Use config files to manage server processes and configurations
  • Setting up an external RDS database and connect to it using environment variables
  • Configuring a custom domain name and setting up DNS
  • Adding SSL termination on the load balancer
  • Adding a http to https rewrite rule to Apache on the web server using a config file


How a desire for control can hurt your security performance

Lately we have seen a lot of focus on security in social media – professionals, companies, organizations trying to increase security awareness. A lot of the information out there is about “control” and “compliance”. The downside of a risk management regime based on strict rules, controls and compliance measures has been demonstrated again and again throughout history, and I’ve also written about it before in terms of getting users on board with the security program. My background is from the oil and gas industry – an industry that has seen several horrific accidents. Two of the more well-known and dramatic ones are the Piper Alpha accident 30 years ago this year, and the Deepwater Horizon blowout in 2010. In both of these cases, investigations pointing to “root causes” concluded with a degraded safety culture, lack of attention to real risks and partially blamed a prescriptive approach to safety. The same arguments are equally valid for security incidents. The goal should be to find a balance between security and operational flow.

acrobatics action balance ballet
Balancing security against performance is necessary to operate with flow and still be at trusted business partner.

Some examples of potentially unhelpful helpfulness

If you are looking for security advice online, it is easy to find. A lot of it will tell you to “trust nobody, lock down everything”. From a traditional security point of view this makes sense – but it does not take the risk context into picture, nor does it balance measures against operational needs (such as keeping your store open, or being able to try new things to innovate and create new products).

Here’s Dr. Eric Cole (he knows a lot about security but sometimes I think his advice is a bit draconian)

Change control is important – but gathering a “change control board” for every change you do may be overkill – if you want ot stay “agile” and able to respond to changing demands.

Another common “rule” that actually does make a lot of sense is not to give end-users admin rights to their work computers. But…. needs will vary. If you are trying to develop new technology but your developers have to go through a lot of red-tape to try out a new technology, it will certainly have some ill effects on your teams ability to innovate. On the other hand, giving developers all free reigns in the name of the “sacred innovation gods” is also not a very good idea. The whole thing is about balance.

Risk acceptance and balance

Security controls are often cumbersome for people. Airport security, nightclub bouncers, two-factor authentication, no admin rights. Security leads to limitation of access in a large number of cases. This obviously has a downside when it comes to how fast we can innovate, how quickly we can produce. The benefit is reduced probability and impact of a major incident – and such incidents are very expensive. The amount of security cumbersomeness people are willing to accept and live with will normally depend on how bad it can get if someone hacks you. If your system controls nuclear weapons, a power plant, or perhaps the production at a chemical plant, incidents can cause real disasters leading to financial and environmental ruin, as well as a large number of fatalities. In this case, you will probably accept a lot of security controls to minimize the chance of something like that happening.

On the other hand, if you are selling some new hot service online, you still need people to trust your service, and you need to comply with privacy laws. This means your security must still be good – but you may nevertheless adopt a slightly higher risk acceptance than in the nuclear weapons case.

The trick is to find a good balance between acceptable risk performance, and good operational flow. This in itself will contribute to greater security performance overall, as the human factor side to cyber risk is very large, something that is often undervalued when designing security controls. To do this in a coherent manner we bring you the mighty tool of the…. risk based threat model.

Threat models for a balanced security strategy

A threat model is best made for a specific system or subsystem. The system can be everything from “the company network” to a specific application or a small software component. The thinking remains the same but the details in your model will change. The whole purpose of it is to understand how and Adversary can perform an Action on a Target to achieve an Objective. There are many ways to model this explained in the literature, but we won’t go into details about them here. If you want the details you can search for attack trees, STRIDE, cyber kill chain.

Context. You need to understand 3 things about the risk context:

  • Who are the stakeholders and what is their interest in the system? Owners, employees, users, customers, suppliers, attackers, insiders
  • What does the system itself do?
  • Who are the threat actors? Use threat intelligence to understand how adversaries approach the system and the supply chain you are a part of.

Inventory and data flow. Create a data flow diagram on the architectural level. Include relevant information such as protocols and main technologies. Describe what each asset is used for, and make a list of what data is being processed and transferred. Make trust boundaries visible in your diagram. For the inventory, consider the potential impact of confidentiality, integrity or availability losses.

Abuse cases. Consider how the various processes and data transfer operations can be abused by an adversary with sufficient access. Access can be physical access, stolen credentials, through malware or direct use of software vulnerabilities. The abuse case is your primary tool for understanding how controls can stop the adversary’s actions.

Detection and mitigation. Your system is probably not wide open to attack. List the most important controls you have in place already. The main purpose of this is to check if you are missing something obvious that you probably should be doing to stop attacks.

Evaluate and prioritize. Evaluate the threats according to the estimated risk. Prioritize controls that will help you reduce the risk of unacceptable actions being taken by adversaries to your most important assets and operational capabilities. Make sure you do not over-stretch the organization’s capabilities – focus on what matters the most first.

Thinking through your context and what you value brings you a long way alone, in particular with solid baseline controls. Maintaining a threat model that is kept up to date regularly with new threat intelligence and other context changes also allows you to ensure you do not fall behind how the world moves. Taking risks is fine, but know what risks you can afford to take – when you do that, you can choose the point for balancing security and performance.





How to manage risk and security when outsourcing development

Are you planning to offer a SaaS product, perhaps combined with a mobile app or two? Many companies operating in this space will outsource development, often because they don’t have the right in-house capacity or competence. In many cases the outsourcing adventure ends in tears. Let’s first look at some common pitfalls before diving into what you can do to steer the outsourced flagship clear of the roughest seas.

Common outsourcing pitfalls

I’ve written about project follow-up before, and whether you are building an oil rig or getting someone to write an app for you, the typical “outsourcing pitfalls” remain the same:

  • Weak follow-up
  • Lack of documentation requirements
  • Testing is informal
  • No competence to ask the right questions
  • No planning of the operations phase
  • Lack of privacy in design

Weak follow-up: without regular follow-up the sense of commitment can get lost for the service provider. It is also increasing the chances of misunderstandings by several magnitudes. If I write a specification of a product that should be made, and even if that specification is wonderfully clear to me, it may be interpreted differently by a service provider. With little communication underway towards the product, there is a good chance the deliverable will not be as expected – even if the supplier claims all requirements have been met.

Another big mistake by not having a close follow-up process, is lost opportunities in the form of improvements or additional features that could be super-useful. If the developer gets a brilliant idea, but has no one to approve of it, it may not even be presented to you as the project owner. So, focus on follow-up – if not you are not getting the full return on your outsourcing investment.

Lack of documentation requirements: Many outsourcing projects follow a common pattern: the project owner writes a specification, and gets a product made and delivered. The outsourcing supplier is then often out of the picture: work done and paid for – you now own the product. The plan is perhaps to maintain the code yourself, or to hire an IT team with your own developers to do that. But…. there is no documentation! How was the architecture set up, and why? What do the different functions do? How does it all work? Getting to grips with all of that without proper documentation is hard. Really hard. Hence, putting requirements to the level of documentation into your contracts and specifications is a good investment with regards to avoiding future misunderstandings and a lot of wasted time trying to figure out how everything works.

Informal or no testing: No testing plan? No factory acceptance test (FAT)? No testing documentation? Then how do you determine if the product meets its quality goals – in terms of performance, security, user experience? The supplier may have fulfilled all requirements – because testing was basically left up to them, and they chose a very informal approach that only focuses on functional testing, not performance, security, user experience or even accessibility. It is a good idea to include testing as part of the contract and requirements. It does not need to be prescriptive – the requirement may be for the supplier to develop a test plan for approval, and with a rationale for the chosen testing strategy. This is perhaps the best way forward for many buyers.

No competence to ask the right questions: One reason for the points mentioned so far being overlooked may be that the buying organization does not have the in-house competence to ask the right questions. The right medicine for this may not be to send your startup’s CEO to a “coding bootcamp”, or for a company that is primarily focused on operations to hire its in-house development team – but leaving the supplier with all the know-how leaves you in a very vulnerable position, almost irrespective of the legal protections in your contract. It is often money well spent to hire a consultant to help follow-up the process – ideally from the start so you avoid both specification and contract pitfalls, and the most common plague of outsourcing projects – weak follow-up.

No planning of operations: If you are paying someone to create a SaaS product for you – have you thought about how to put this product into operation? Often important things are left out of the discussion with the outsourcing provider – even if their decisions have a very big impact on your future operations. Have you included the following aspects into your discussions with the dev teams:

  • Application logs: what should be logged, and to what format, and where should it be logged?
  • How will you deploy the applications? How will  you mange redundancy, content delivery?
  • Security in operations: how will you update the apps when security demands it, for example through the use of dependencies/libraries where security holes become known? Do you at all know what the dependencies are?
  • Support: how should your applications be supported? Who picks up the phone or answers that chat message? What information will be available from the app itself for the helpdesk worker to assist the customer?

Lack of privacy in design: The GDPR requires privacy to be built-in. This means following principles such as data minimization, using pseudonomization or anonymization where this is required or makes sense, means to detect data breaches that may threaten the confidentiality and integrity (and in some cases availability) of personal information. Very often in outsourcing projects, this does not happen. Including privacy in the requirements and follow-up discussions is thus not only a good idea but essential to make sure you get privacy by design and default in place. This also points back to the competence bit – perhaps you need to strengthen not only your tech know-how during project follow-up but also privacy and legal management?

A simple framework for successful follow-up of outsourcing projects

The good news is that it is easy to give your outsourcing project much better chances of success. And it is all really down to common sense.

Activities in three phases for improving your outsourcing management skills


First, during preparation you will make a description of the product, and the desired outcomes of the outsourcing project. Here you will have a lot to gain from putting in more requirements than the purely functional ones – think about documentation, security, testing and operations related aspects. Include it in your requirements list.

Then, think about the risk in this specification. What can go wrong? Cause delays? Malfunction? Be misunderstood? Review your specification with the risk hat on – and bring in the right competence to help you make that process worthwhile. Find the weaknesses, and then improve.

Decide how you want to follow-up the vendor. Do you want to opt for e-mailed status reports once per week? The number of times that has worked for project follow-up is zero. Make sure you talk regularly. The more often you interact with the supplier, the better the effect is on quality, loyalty, and priorities. Stay on the top priority list for your supplier – if not your product will not be the thing they are thinking about when coming to the office in the morning. Things you can do to get better project follow-up:

  • Regular meetings – in person if you are in the same location, but also on video works well.
  • Use a chat tool such as Slack, Microsoft Teams or similar for daily discussions. Keep it informal. Be approachable. That makes everything much better.
  • Always focus on being helpful. Avoid getting into power struggles, or a very top-down approach. It kills motivation, and makes people avoid telling you about their best ideas. You want those ideas.

Competence. That is the hardest piece of the pussle. Make sure you take a hard look at your own competence, and the competence you have available before deciding you are good to go. This determines if you should get a consultant or hire someone to help follow-up the outsourcing project. For outsourcing of development work, rate your organization’s competence within the following areas:

  • Project management (budgets, schedule, communications, project risk governance, etc)
  • Security: do you know enough to understand what cyber threats you need to worry about during dev, and during ops? Can you ask the right questions to make sure your dev team follows good practice and makes the attack surface as small as it should be?
  • Code development: do you understand development, both on the organizational and code level? Can you ask the right questions to make sure good practice is followed, risks are flagged and priorities are set right?
  • Operations: Do you have the skills to follow-up deployment, preparations for production logging, availability planning, etc?
  • User experience: do you have the right people to verify designs and user experiences with respect to usability, accessibility?
  • Privacy: do you understand how to ensure privacy laws are followed, and that the implementation of data protection measures will be seen as acceptable by both data protection authorities and the users?

For areas where you are weak, consider getting a consultant to help. Often you can find a generalist who can help in more than one area, but it may be hard to cover them all. It is also OK to have some weaknesses in the organization, but you are much better off being aware of them than running blind in those areas. The majority of the follow-up would require competence in project management and code development (including basic security), so that needs to be your top priority to cover well.

Work follow-up

Now we are going to assume you are well-prepared – having put down good requirements, planned on a follow-up structure and that you more or less have covered the relevant competence areas. Here are some hints for putting things into practice:

  • Regular follow-up: make sure you have formal follow-up meetings even if you communicate regularly on chat or similar tools. Make minutes of meetings that is shared with everyone. Make sure you make the minutes – don’t empower the supplier to determine priorities, that is your job. The meetings should all be called for with agendas so people can be well prepared. Here are topics that should be covered in these meetings:
    • Progress: how does it look with respect to schedule, cost and quality
    • Ideas and suggestions: useful suggestions, good ideas? If someone has a great idea, write down the concept and follow-up in a separate meeting.
    • Problems: any big issues found? Things done to fix problems?
    • Risks: any foreseeable issues? Delays? Security? Problems? Organizational issues?
  • Project risk assessment: keep a risk register. Update it after follow-up meetings. If any big things are popping up, make plans for correcting it, and ask the supplier to help plan mitigations. This really helps!
  • Knowledge build-up: you are going to take over an application. There is a lot to be learned from the dev process, and this know-how often vanishes with project delivery. Make sure to write down this knowledge, especially from problems that have been solved. A wiki, blog, and similar formats can work well for this, just make sure it is searchable.
  • Auditing is important for all. It builds quality. I’ve written about good auditing practices before, just in the context of safety, but the same points are still valid for general projects too: Why functional safety audits are useful.


  • Make sure to have a factory acceptance test. Make a test plan. This plan should include everything you need to be happy with to say you will take it over:
    • Functions working as they should
    • Performance: is it fast enough?
    • Security: demonstrate that included security functions are working
    • Usability and accessibility: good standards followed? Design principles adhered to?
  • Initial support: the initial phase is when you will discover the most problems – or rather, your users will discover them. Having a plan for support from the beginning is therefore essential. Someone needs to pick up the phone or answer that chat message – and when they can’t, there must be somewhere to escalate to, preferably a developer who can check if there is something wrong with the code or the set-up. This is why you should probably pay the outsourcing supplier to provide support in the initial weeks or months before you have everything in place in-house; they know the product best after making it for you.
  • Knowledge transfer: the developers know the most about your application. Make sure they help you understand how everything works. During the take-over phase make sure you ask all questions you have, that you have them demo how things are done, take advantage of any support contracts to extend your knowledge base.

This is not a guarantee for success – but your odds will be much better if you plan and execute follow-up in a good manner. This is one way that works well in practice – for all sorts of buyer-supplier relationship follow-up. Here the context was software – but you may use the same thinking around ships, board games or architectural drawings for that matter. Good luck with your outsourcing project!

Comments? They are very welcome, or hit me up on Twitter @sjefersuper!


Do you consider security when buying a SaaS subscription?

tl;dr;  SaaS apps often have poor security. Before deciding to use one do a quick security review. Read privacy statements, ask for security docs, and test authentication practices, crypto and console.log information leaks before deciding if you want to trust the app or not. This post gives you a handy checklist to breeze through your SaaS pre-trial security review.

Over the last year I’ve been involved in buying SaaS access for lots of services in a corporate setting. My stake in the evaluation has been security. Of course, we can’t do active attacks or the like to evaluate if we are going to buy a software, but we can read privacy statements, look for strange things in the HTML and test their login process if they allow you to set up a free account or a trial account (most of them do). Here’s what I’ve generally found of challenges:

  • Big name players know what they are doing, but they have so many options for setup that you need to be careful what you select, especially if you have specific compliance needs.
  • Smaller firms (including startups)  don’t offer a lot of documentation, and quite often when talking to them they don’t really have much in place in terms of security management (they may still make great software, it is just harder to trust it in an enterprise environment)
  • Authentication blunders are very common. From HR support systems to payroll to IT management tools, we’ve found a lot of bad practices:
    • Ineffective password policies (5 digits and digits only, anyone?)
    • A theoretical password policy in place but validation is only performed client-side (that means it is easy to trick the server into setting a very weak password, or even no password in same cases, if you should so desire)
    • Lack of cookie security for session cookies (missing HTTPOnly and Secure flags, allowing for cookie theft by XSS attacks or man-in-the-middle attacks)
    • Poor password reset processes. In one case I found the supposedly random string used for a password reset link to be a base 64 encoding of my username…. how hard is that to abuse?
  • When you ask about development practices and security testing, many firms will lack such processes but try to explain that they implicitly have them because their developers are so smart (even with the authentication blunders mentioned above).

Obviously, at some point, security will be so bad that you have to say “No, we cannot buy this shiny thing, because it is rotten on the inside”. This is not a very popular decision in some cases, because the person requesting the service probably has some reason to request it.

egg power fear hammer
Don’t put your data and business at risk by using a SaaS product you can’t trust. By doing a simple due dilligence job you can at least identify services you definitely should be avoiding!

In order to evaluate SaaS security without spending too much time on it I’ve come up with a process I find works pretty well. So, here’s a simple way to sort the terrible from the average!

  1. Read the privacy statement and the terms and conditions. Not the whole thing, that is what lawyers are for (if you have some), but scan it and look for anything security related. Usually they will try to explain how they protect your personal data. If it is a clear and understandable explanation, they usually know what they are doing. If not, they usually don’t.
  2. Look for security or compliance documentation, or request it from their sales/support team. Specific questions to consider are:
    1. Do they offer encryption at rest (if this is reasonable to expect)?
    2. Do they explain how they store passwords?
    3. Do they use trustworthy data centers, and have some form of compliance proof for good practice for their data centers? SOC-2 reports, ISO certificates, etc?
    4. Do they guarantee limited access for insiders?
    5. Do they explain what cryptographic controls they are using for TLS, signatures and at-rest encryption? That means how they perform key management and exchange, what ciphers they allow, and the key strength they require?
    6. Do they say anything about vulnerability management and security testing?
    7. Do they say anything about incident handling?
  3. Perform some super-lightweight testing. Don’t break the law but do your own due diligence on the app:
    1. Create a free account if possible, and test if the authentication practices seem sound:
      • Test the “I forgot my password functionality” to see if the password reset link has an expiry time, and if it is a unguessable” link
      • Try changing your password to something really bad, like “password” or “dog”. Try to replay post requests if stopped by client-side validation (this can be done directly from the dev tools in Firefox, no hacker tool necessary)
      • Try logging in many times with the wrong password to see what happens (warning, lockout,… or most likely, nothing)
    2. Check their website certificate practices by running sslscan, or by using the online version at If they support really weak ciphers, it is an indicator that they are not on the ball. If they support SSL (pre TLS 1.0) it is probably a completely outdated service.
    3. Check for client-side libraries with known vulnerabilities. If they are using ancient versions of jQuery and other client side libraries, they are likely at risk of being hacked. This goes for WordPress sites as well, including their plugins.
    4. Check for information leaks by opening the console (Ctrl + I in most browsers). If the console logs a lot of internal information coming from the backend, they probably don’t have the best security practices.

So, should you dump a service if any of these tests “fail”? Probably not. Most sites have some weak points, and sometimes that is a tradeoff for some reason that may be perfectly legitimate. But if there are a lot of these “bad signs” in the air and you would be running some critical process or storing your company secrets in the app, you will be better off finding another service to suit your needs – or at least you will be able to sleep better at night.


When society breaks down: how do we respond?

Consider this: internet is down. Power is out. And the water in the tap is no longer safe to drink. The stores are basically out of groceries. And the banking sector is not working. No mobile payments. No credit cards accepted. And no ATM’s are working. Scenarios like this may be dystopia but are perhaps less far-fetched today than a few years ago. Some recent reports hinting of this have come out of the Ukrainian conflict as well as more recent events of cyber attacks targeting the utility sectors in the United States, Europe and the Middle East. There is no other way to put it than this: we are as a society vulnerable.

action active activity adult
When social functions break down, a failure of businesses and organizations to provide basic services make the situation even more difficult ot cope with from the individual to the government level.

Sweden is taking steps to increase the population’s preparedness for a major crisis, up to and including invasion by a foreign power. Norwegian authorities are planning a similar move. This type of communication was common during the cold war but feels chilling today. We are no longer used to thinking about disasters that target society at this scale.

A major conflict today would for sure include cyber domain operations, and most likely not only for information gathering. Availability of key services would be hit, and this could lead to power outages, water supply failure and payment system collapse. How do we cope in this situation?

Most people are not prepared for the “usual channels” to be unavailable. Most organizations are unprepared for disasters like this. This further exacerbates the challenges individuals would be faced with in the event of a crisis, because many businesses are essential for providing services and goods. When these businesses cannot deliver, it means power is unavailable, hospitals close, food is not available in the store and the fancy autonomous public transport systems grind to a halt.

Because of this, it is a civic duty for businesses to plan not only for a rainy day, but for long-term hurricane conditions. When the economy fails to produce the services and goods people depend on, we all suffer. Here are five bullet points for building resilience from the individual level, to our workplaces, and to society as a whole.

  1. Do like the Swedes: keep an emergency supply of food, water, and other necessities at home. Have a plan for how to act in the case of a crisis.
  2. At the workplace, do not stop at a risk assessment for “normal operations”. Identify business continuity challenges, and abnormal situations that can occur, including natural disasters, nationwide cyber attacks, terror attacks and a state of war. What services should the organization be able of supplying under such conditions? How can a plan be put in place to be able of doing so?
  3. Planning is smart, but without training its value is very limited. This is why businesses run stress tests, table-top exercises, red-team simulations and the like. We do, however, focus on risks under “normal conditions”. Have you tested your business continuity handling plan the same way? You probably should. Exercise emergency response with no network access, with phone lines down and your staff dispersed.
  4. Do not get paranoid, but also do not be afraid to mention what people typically would see as “black swans”. Only by acknowledging that disasters do happen, we can prepare to restore functionality to the level we have defined necessary.
  5. Engage in conversations and organizations that keep you on top of societal risks, and how you can contribute. Contributing to the security of society as a whole is the essence of social corporate responsibility.

If we keep contributing during a crisis, we will increase our collective ability to handle adversity. This is why business continuity needs to be part of our thinking around social corporate responsibility.


How to build emergency preparedness for cybersecurity incidents

Business continuity and emergency preparedness have become familiar concepts for many businesses – and having such risk management practices in place is expected in many industries. In spite of this, apart from software companies, inclusion of cybersecurity and preparing for handling of serious cyber attacks and security incidents is far from mature. Many businesses have digitized their value chains to a very high degree without thinking about how this affects their overall risk picture. Another challenge for many businesses that have seen the need to include their digital footprint in their risk management process, is that they don’t know where to start. That is what this post is about: how do you start to think about emergency preparedness for cyber incidents? If you have a robust process for this in place, this post is not meant for you. This is a “how-to” for those who stand bewildered at the start position of their crisis management planning process.

You need a clear plan and a trained crew for efficient cyber incident response.

Know what you have

Before planning your incident response, or emergency preparedness plan, you should have a clear overview of what assets you have that is worth protecting. Creating a detailed asset inventory can be a daunting task. However, for most organizations, it is sufficient to identify the key information and organizational assets without aiming for completeness.

  • What are your main business processes? Identify all of the main processes you need to work in order for your organization to serve its purpose. The breakdown can be of different granularity, but here’s an example for an e-commerce business:
    • Management and leadership
    • Sales and marketing
    • Procurement and logistics
    • Software development
    • Customer support
    • Accounting
  • For each of the main business processes you have, there will be various types of assets that are necessary to make that process work. Think about what you need in various categories:
    • Key personnel
    • Software you need to get the job done
    • Data that is needed to support the function (if you know what software you depend on, this is often easier to identify)

Why are we mentioning people here? Often people have knowledge that cannot easily be replaced within the organization, or that would require considerable effort and investment. If such a person disappears, the situation can be hard to deal with. That is why, also from an information security point of view, it is important to know who your key employees are, and put down a plan for what to do if they are not available.

When you have identified these assets, it is a good idea to group them into two categories: critical and non-critical (you can use more than 2 categories if you want to, but a binary division is usually sufficient). Critical assets would lead to serious consequences if the security is breached: if data is leaked, or changed in an unauthorized manner, or made unavailable. It is unfortunate if non-critical assets are breached too, but not at a level where the business itself can be threatened. The critical assets are your crown jewels – the assets you need to protect as good as you can.

Baseline defense: do the small things that matter

Before planning how to respond to a cyber attack, we should introduce some baseline practices that do not depend on criticality or risk assessments. These are practices all organizations should aim to internalize; they significantly reduce the likelihood that a cyber attack would be successful, and they also prepare you to respond to an attack when it happens.

  • Introduce a security policy and make it known to the organization. Work systematically to make sure the policy is adhered to.
  • Maintain the data register (using the process described above for “knowing what you have”). This way you make sure critical assets do not get overlooked.
  • Include security requirements when selecting suppliers. Do not get breached because a supplier or business partner has weak security practices.
  • Take regular backups of all critical data. This way, you can restore your data if they should become unavailable or destroyed, whether this happens because of a hacker’s malicious actions or due to a hardware failure.
  • Use firewall rules to deny all traffic that is not needed in your business. Deny all incoming requests, unless there is a specific reason to keep a service available.
  • Run up-to-date endpoint protection such as antivirus software on all computers.
  • Keep all of your software up-to-date and patched. Do not forget appliances and IoT devices.
  • Do not give end users administrative access to their computers.
  • Give security awareness training to all employees.

With this in place, 80% of the job is done. Now you can focus on the “disaster scenarios”; those where you crown jewels are at risk.

Prepare to defend your assets

You know what assets you have. You know what your crown jewels are. You have your baseline security in place. Now you are ready to take on the remaining risk – responding to attacks and more advanced incidents. Here’s how you prepare for that.

Threat modeling

Before you develop your incident response plan, it pays off to create a simple threat model. Your model should describe credible attack patterns. In order to identify such attack patterns, you should think about who the attacker would be, and what their motivation would be. Is it a script kiddie, a person without deep technical knowledge hacking for fun using tools downloaded from the internet? Is it a cyber crime group hoping to earn money on extortion or by selling your intellectual property? Is it a nation-state actor, hoping to use your company as a foothold for attacking government assets? Or perhaps it is an insider threat, a dishonest or angry employee attacking his own employer? Likely scenarios depend on your assumptions here.

You don’t need a very detailed threat model to gain understanding that can aid your incident response planning. You should think about phases of the attack?

  • How is the initial breach obtained? In most cases this would be some form of social engineering, like phishing.
  • How do they get a foothold and gain persistence? Malware based? Using built-in functions?
  • How do they get access to the crown jewels? What actions will they perform on the object?
  • What are the consequences of the attack for your organization and its stakeholders?

Having this down, you should start to prepare an incident response plan. Thinking about this in phases too is helpful:

  • Preparation
  • Incident detection and escalation
  • Containment
  • Eradication
  • Recovery
  • Lessons learned

During preparation you should get down who is responsible for incident handling, who should be communicated with and how suspected incidents should be reported. Include a budget for training and running exercises. Cyber threat incident response needs to be tested the same way we do fire drills.

Incident detection is difficult. Various reports all indicate the average time from compromise to detection of advanced attacks is somewhere between 3 months and 2 years. There are many ways to detect that something is wrong:

  • A user notices strange behavior of lack of access
  • Monitoring of logs and security systems may report unusual signals
  • A hacker contacts you for a ransom or to state demands

In all cases the company should have a clear process for categorizing potential incidents, verifying if it is a real incident or not, and making a decision to start incident response.

Containment is about stopping the problem from spreading throughout the network, and gathering evidence. Be aware that cutting access to the internet can sometimes set off pre-programmed destructive routines. Therefore containment should be based on observation of the hacker behavior within the network on a case by case basis.

Eradication is about removing the problem: taking away the persistent access, removing malware, patching security holes. The right way to do this is to format all disks, clean all data, and then restore from original media and trusted backups.

Recovery is about getting back to business: recovering the service at an acceptable level. It is not uncommon to see malware reappear after recovery, so testing in a controlled environment is always good practice, before connecting the restored system to the business network again.

Lessons learned is important. In this phase an after action review is done: how could this happen, what was the reason? Do a root cause analysis. Summarize what worked well in response, and what did not. Make recommendations for changes in practice or policy – and follow up on it.

If you have this down: knowing what your crown jewels are, a solid baseline security system and a risk based incident response plan your organization will be much more robust than before. The risk exposure of your organization to cyber threats will be greatly reduced – but do not forget that security is a continuous process: as the threat landscape changes, your security management should too. This is why you need to maintain your threat model, and update your response plan.

Packaging a Node app for Docker – from Windows

Container technologies are becoming a cornerstone of development and deployment in many software houses – including where I have my day job. Lately I’ve been creating a small web app with lots of vulnerabilities to use for security awareness training for developers (giving them target practice for typical web vulnerabilities). So I started thinking about the infrastructure: packing up the application in one or more containers – what are the security pitfalls? The plan was to look at that but as it turned out, I struggled for some time just to get the thing running in a Docker container.

First of all, the app consists of three architectural components:

  • A MongoDB database. During prototyping I used a cloud version at That has worked flawlessly.
  • A Vue 2.0 based frontend (could be anything, none of the built-in vulnerabilities are Vue specific)
  • An Express backend primarily working as an API to reach the MongoDB (and a little sorting and such)

So, for packing things up, I started with taking the Express backend and wanting to add that to a container to run with Docker. In theory, the container game should work like this:

  1. Create your container image based on a verified image you can download from a repository, such as Docker Hub. For node applications the typical recommendation you will find in everything from Stack Overflow to personal blogs and even official doc pages from various projects is to start with a Node image from Docker Hub.
  2. Run your docker image using the command
    docker run -p exposedIP:hostIP myimage
  3. You should be good to go – and access the running NodeJS app at localhost:hostIP

So, when we try this, it seems to run smoothly…. until it doesn’t. The build crashes – what gives?

After some more googling we tried to use node:8-alpine as the base image. Didn’t work, it cannot install the necessary build tools to run libxmljs, with a warning that a required file is not available for the Alpine package manager.

Building on top of Alpine, a minimal Linux distribution popular for use in containers in order to reduce the image size, we try to install some OS specific build tools required in order to install the npm package libxmljs. This package is a wrapper for the xmllib2 library for C (part of the Gnome project). Because that is what it is, it needs to set up those bindings locally for the platform it is running on, hence it needs a C compiler and a version of Python 2.7 to make this happen. To install packages on Alpine one uses the apk package manager. These packages are obviously there, so why does it fail?

Normally building a NodeJS application for production would involve putting the package.json file on the production environment and running npm install. The actual JavaScript files are not transferred (stored on the folder node_modules), they are fetched from their sources. When installing modules that need to hook into platform specific resources, this is reflected in the contents of the local node module after first installation. So if you copy your node_modules folder over to the container, this can fail. In my case it did: the app was developed on a Windows 10 computer, and we were trying to install it now on Alpine Linux in the container. The image was built with the local dev files copied to the app directory of the container image: and I had not told it what not to copy. Here’s the Dockerfile:

EDIT: use node:8 official image, not alpine, as it does not play well with glibc dependencies (such as libxml2).

FROM mhart/alpine-node:8
FROM node:8

COPY . .

# Fixing dependencies for node-gyp / libxmljs
RUN apk add –no-cache make gcc g++ python
RUN apt-get install make gcc g++ python

RUN npm install –production

CMD [“node”, “index.js”]

After adding the “no-cache” option on the apk command the libraries installed fine. But running the container still led to crash.

The error message we got when running the container was “Error loading shared library…. Exec format error”. This is because shared library calls are platform specific and built into the built version of libxmljs in node_modules.

After a few cups of coffee I found the culprit: I had copied the node_modules folder from my Windows working folder. Not a good idea. So, adding a .dockerignore file before building the image fixed it. That file includes this:


The backlog file is just a debug log. After doing this, and building again: Success!

Now running the image with

docker run -p 9000:9000 -d my_image_name

gives us a running container that serves the Exposed port 9000 to the localhost port 9000. I can check this in my browser by going to localhost:9000


OK, so we’re up and running with the API. Next tasks will be to set up separate containers for the frontend and possibly for the database server – and to set up proper networking between them. Then we can look at how many configuration mistakes we have made, perhaps close a few, and be ready to start attacking the application (which is the whole purpose of this small project).