Running an automated security audit using Burp Professional

Reading about hacking in the news can make it seem like anyone can just point a tool at any website and completely take it over. This is not really the case, as hacking, whether automated or manual, requires vulnerabilities.

A well-known tool for security professionals working with web applications is Burp from Portswigger. This is an excellent tool, and comes in multiple editions from the free community edition, which is a nice proxy that you can use to study HTTP requests and responses (and some other things), to the professional edition aimed at pentesting and enterprise which is more for DevOps automation. In this little test we’ll take the Burp Professional tool and run it using only default settings against a target application I made last year. This app is a simple app for posting things on the internet, and was just a small project I did to learn how to use some of the AWS tools for deployment and monitoring. You find it in all its glory at https://www.woodscreaming.com.

Just entering the URL http://www.woodscreaming.com and launching Burp to attack the application first goes through a crawl and audit of unauthenticated routes it can find (it basically clicks all the links it can find). Burp then registers a user, and starts probing the authenticated routes afterwards, including posting those weird numerical posts.

Woodscreaming.com: note the weird numerical posts. These are telltale signs of automated security testing with random input generation.

What scanners like Burp are usually good at finding, is obvious misconfigurations such as missing security headers, flags on cookies and so on. It did find some of these things in the woodscreaming.com page – but not many.

Waiting for security scanners can seem like it takes forever. Burp estimated some 25.000 days remaining after a while with the minimal http://www.woodscreaming.com page.

After runing for a while, Burp estimated that the remaining scan time was something like 25.000 days. I don’t know why this is the case (not seen this in other applications) but since a user can generate new URL paths simply by posting new content, a linear time estimation may easily diverge. A wild guess at what was going on. Because of this we just stopped the scan after some time as it was unlikely to discover new vulnerabilities after this.

The underlying application is a traditional server-driven MVC application running Django. Burp works well with applications like this and the default setup works better than it typically does for single page applications (SPA’s) that many web applications are today.

So, what did Burp find? Burp assigns a criticality to the vulnerabilities it finds. There were no “High” criticality vulns, but it reported some “Medium” ones.

Missing “Secure” flag on session cookies?

Burp reports 2 cookies that seem to be session cookies and that are missing the Secure flag. This means that these cookies would be set also if the application were to be accessed over an insecure connection (http instead of https), making a man-in-the-middle able to steal the session, or perform a cross-site request forgery attack (CSRF). This is a real find but the actual exposure is limited because the app is only served over https. It should nevertheless be fixed.

A side note on this: cookies are set by the Django framework in their default state, no configuration changes made. Hence, this is likely to be the case also on many other Django sites.

If we go to the “Low” category, there are several issues reported. These are typically harder to exploit, and will also be less likely to cause major breaches in terms of confidentiality, integrity and availability:

  • Client-side HTTP parameter pollution (reflected)
  • CSRF cookie without HTTPOnly flag set
  • Password field with autocomplete enabled
  • Strict transport security not enforced

The first one is perhaps the most interesting one.

HTTP paramter pollution: dangerous or not?

In this case the URL parameter reflected in an anchor tag’s href attribute is not interpreted by the application and thus cannot lead to bad things – but it could have been the case that get parameters had been interpreted in the backend, making it possible to have a person perform an unintended action in a request forgery attack. But in our case we say as the jargon file directs us: “It is not a but, it is a feature”!

So what about the “password field with autocomplete enabled”? This must be one of the most common alerts from auditing software today. This can lead to unintended disclosure of passwords and should be avoided. You’ll find the same on many well-known web pages – but that does not mean we shouldn’t try to avoid it. We’ll put it on the “fix list”.

Are automated tests useful?

Automated tests are useful but they are not the same as a full penetration test. They are good for:

  1. Basic configuration checks. This can typically be done entirely passively, no attack payloads needed.
  2. Identifying vulnerabilities. You will not find all, and you will get some false positives but this is useful.
  3. Learning about vulnerabilities: Burp has a very good documentation and good explanations for the vulnerabilities it finds.

If you add a few manual checks to the automated setup, perhaps in particular give it a site-map before starting a scan and testing inputs with fuzzing (which can also be done using Burp) you can get a relatively thorough security test done with a single tool.

Defending against OSINT in reconnaissance?

Hackers, whether they are cyber criminals trying to trick you into clicking a ransomware download link, or whether they are nation state intelligence operatives planning to gain access to your infrastructure, can improve their odds massively through proper target reconnaissance prior to any form of offensive engagement. Learn how you can review your footprint and make your organization harder to hack.

https://cybehave.no

Cybehave has an interesting post on OSINT and footprinting, and what approach companies can take to reduce the risk from this type of attack surface mapping: https://cybehave.no/2019/03/05/digital-footprint-how-can-you-defend-against-osint/ (disclaimer: written by me and I own 25% of this company).

tl;dr – straight to the to-do list

  • Don’t publish information with no business benefit and that will make you more vulnerable
  • Patch your vulnerabilities – both on the people and tech levels
  • Build a friendly environment for your people. Don’t let them struggle with issues alone.
  • Prepare for the worst (you can still hope for he best)

Storing seeds for multifactor authentication tokens

When setting up an application to use two-factor authentication for example with Google Authenticator, each user will have a unique seed value for the authenticator. The identity server will require knowledge of the seed to verify the token – meaning you will have to store it and retrieve it somehow. This means that if an attacker gets access to the storage solution that links OTP secret seeds to user ID’s (e.g. usernames), the protocol is broken. So, trying to think up some options for securing the secrets – we cannot hash and salt it because it breaks the OTP authentication flow. We are hence left with encrypting the seed before storing it.

The most practical seems to be a symmetric crypto approach, the question is what to use as the crypto key. Here are some approaches I’ve seen people discuss that all seem bad: 

  • User password: if you can phish the password, then you can also generate the OTP provided you know which algorithm/library is used
  • A static application secret: should be safe provided that secret is never leaked but using a static secret means that if it is compromised, all users are compromised. Still better than the user password, though. 
  • Using non-static user level meta data to create a unique key for each user that is not vulnerable to phishing or guessing. Typically visible to admins.
Get username/password
Verify username/password
Get OTP seed (encrypted)
Get metadata and reconstruct encryption key
Verify OTP
Authenticate user and store timestamp and other auth metadata
Construct new encryption key
Encrypt seed
Store in database

The question is what metadata to use. We need the following properties to be true:

  • Not possible to guess for a third party even if we tell what metadata it is
  • Not possible to reconstruct for an administrator with access to the account
  • Not possible to phish or obtain through social engineering or client side attacks

There are many possibilities but here is one possible solution that would satisfy all the above requirements:

Key = Password (Not available to admins) + Timestamp for last login (not guessable/phishable)

Deploying Django to app engine with Python 3.7 runtime – fails because it can’t find pip?

I had an interesting error that took quite some time to hunt down today. These are basically some notes on what caused it and how I tracked it down. I have an app that is deployed to Google App Engine standard. It is running Django and using the Python 3.7 runtime – and it was working quite well for some time. Then yesterday I was going to deploy an update (actually just adding some CSS tweaks), it failed, with a cryptic error message. Running “gcloud app deploy” lead to the following error message:

File upload done.
Updating service [default]…failed.
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build a33ff087-0f47-4f18-8654-********* status: FAILURE.
Error ID: B212CE0B.
Error type: InternalError.
Error message: pip_install_from_wheels had stderr output:
/env/bin/python3.7: No module named pip
error: pip_install_from_wheels returned code: 1.

This is weird: this is a normal Python project using a requirements.txt file for its dependencies. The file was generated using pip freeze, and should not contain anything weird (it doesn’t). Searching the wisdom of the internet reveals that nobody else seems to have this problem, and it only occurred since yesterday. My hunch was that they’ve changed something on the GAE environment that broke something. Searching the net gives us these options:

  • The requirements.txt file has weird encoding and contains Chinese signs/letters? That was not it.
  • This is because you need to install some special packages for using Python3.. was also not the case and would have been weird changing since a few days ago…
  • You need to manually install pip to make it work – which may be the case sometimes but without SSH access to the instance this isn’t obvious how to do.
The trick is often looking at the right logs….

So, being clueless I turned to looking for the right logs to figure out what is going on. Not being an expert on the GAE environment led to some hunting in the web console until I found “Cloud build”, which sounded promising. That was the right place to be – GAE uses a build process in the cloud to first build the application, and then a Docker image to push to the internal Google Cloud Docker repository. Hunting the build log for this finds this little piece of gold:

Step #1 - "builder": INFO pip_install_from_wheels took 0 seconds
Step #1 - "builder": INFO starting: pip_install_from_wheels
Step #1 - "builder": INFO pip_install_from_wheels /env/bin/python3.7 -m pip install --no-deps --prefix /tmp/tmp9Y3aD7/env /tmp/tmppuvw4s/wheel/pip-19.0.1-py2.py3-none-any.whl --disable-pip-version-check
Step #1 - "builder": INFO `pip_install_from_wheels` stdout:
Step #1 - "builder": Processing /tmp/tmppuvw4s/wheel/pip-19.0.1-py2.py3-none-any.whl
Step #1 - "builder": Installing collected packages: pip
Step #1 - "builder": Found existing installation: pip 18.1
Step #1 - "builder": Uninstalling pip-18.1:
Step #1 - "builder": Successfully uninstalled pip-18.1
Step #1 - "builder": Successfully installed pip-19.0.1
Step #1 - "builder": Failed. No module /env/python/pip

Before the weird error we see it is trying to uninstall pip-18.1, then install pip-19.0.1 (a more recent version), and then it can’t find pip afterwards and the build process fails. This has not been configured by me and is probably Google configuring automatic upgrades of some packages during build – and here it breaks the workflow.

Fixing it

The temporary fix was simple. Adding “pip==18.1” to the requirements.txt file, allowed the build process to run and it deployed nicely.

What did we learn from this?

  • API tools give only partial error messages, making debugging hard.
  • Automated upgrade configs are good but can cause things to break in unforeseen ways.
  • Finding the right logs is the key to fixing weird problems

Combining VueJS and Django to build forms with custom widgets

This post is brief and explains a pattern that may be dangerous but still is very handy for combining VueJS with Django templates for dynamic forms. Here’s the case: I need to build a form for sending out some messages. One of the form widgets is a <select> tag where each <option> is a model instance from Django. The widget will then show the name of that model instance in the UI, but this does not provide enough context to be useful, we also need some description text. There are basically two options for how to handle this:

  1. Use the form “as-is” but provide the extra context in the UI by pulling some extra information and building an information box in the UI.
  2. Create a custom widget, and bind it to the Django model form using a hidden field.

Both are probably equally good, but I went with the second option. So here’s what I did:

  1. Build a normal Django model form, but change the widget for the field in question to type “HiddenInput” in the form.py file.
  2. Build a selector widget using VueJS that allows the user to get the desired content and review the various options with full context (including images and videos, things you can’t put inside a dropdown list. We are binding the selected choices to frontend data using the v-model directive in VueJS.
  3. Set the hidden field to set its value based on the data value stored in the frontend using that v-model directive
  4. Process the form as you normally would with a Django model form.

The form definition remains very simple. Here’s the relevant class from this example:

class MailForm(forms.ModelForm):

    class Meta:
        model = Campaign
        fields = ('name','to','elearning',)
        widgets = {
            'elearning': forms.HiddenInput(attrs={':value': 'module.pk'})
        }

The selector widget can take any form you could desire. The point in this project was to show some more context for the “eLearning” model. The user here gets notification about enrollment in an eLearning module by e-mail. The administrator setting up the program needs to get a bit of context about that e-learning, such as the name of the module, a description of its content, and perhaps a preview of a video or other multimedia. Below is an example of a simple widget of this type. The course administrator can here browse through the various options by clicking next, and the e-mail form is automatically updated.

Of course, to do that binding we need a bit of JavaScript in the Django template. We need to perform the following tasks to make our custom widget work:

  1. Fetch information about all the options from the server. We need to create an API endpoint for this, that can deliver JSON data to the frontend.
  2. Set the data item bound to the Django form based on the user’s current selection

Now the form can be submitted and processed using the normal Django framework patterns – but with a much more context-rich selection widget than a simple dropdown list.

Is it safe to do this?

Combining frontend and server-side rendering with different templates for HTML rendering can be dangerous. See this excellent write-up on XSS vulnerabilities that can be the result from such combinations: https://github.com/dotboris/vuejs-serverside-template-xss.

This is a problem when user input is injected via the server-side template as the user can supply the interpolation tags as part of the input. In our case there is no user input in those combinations. However, if you need to take user input and rerender this using the server-side templates of some framework like Django, here are some things you can do to harden against this threat:

  • Use the v-pre directive in VueJS
  • Sanitize the input to discard unsafe characters, including VueJS delimiters
  • Escape generated output from the database to avoid injections making it as executable JavaScript reaching the user’s context

Security awareness: the tale of the Minister of Fisheries and his love of an Iranian former beauty queen

An interesting story worthy of inspiring books and TV shows is unfolding in Norway. The Minister of Fisheries, Per Sandberg (born 1960), from the Progress Party (a populist right party), spent his summer holiday in Iran together with his new girlfriend, a 28-year old former beauty queen who fled to Norway do escape forced marriage when she was 16. The minister brought his smartphone, where he has access to classified information systems. He forgot to inform the prime minister before after he left, a breach of security protocol. He ignored security advice from the Norwegian security police, responsible for national security issues and counter-intelligence. He is still a member of the cabinet. This post is an attempt at making sense of this, and what the actual risk is. A lot of people in Norway have had their say in media about this case, both knowledgeable voices, and less reasonable ones.

Some context: Norwegian-Iranian relations

Traditionally there has been little trade between Iran and Norway. Recently, following the nuclear agreement between Iran and the US, UK, France, China, Russia and Germany this has started to change. Norway has seen significant potential for exports to Iran of fish and aquaculture technologies. In the last year or so, Minister Sandberg has been central to this development (see timeline further down on Sandberg’s known touch points with Iran).

In the Norwegian public skepticism of the Iranian regime is high, and there has been vocal criticism of establishing trade relationships over human rights concern.

The Norwegian Iranian interest spheres are also intersecting in the Middle East. Iran has established tighter relations with Russia since 2016 when it started to allow Russian bombers to take off from Iranian air force bases for bombing missions inside Syria. The Norwegian-Russian relations are strained, following the response of NATO and the EU to Russian operations in the Ukraine, influencing of western elections and a general intensification of cyber operations against Norwegian targets (see open threat assessment from Norwegian military intelligence: https://forsvaret.no/fakta/undersokelser-og-rapporter/fokus2018/rapport (in Norw.). Operations against Norwegian government officials by Iranian services may thus also be driven by other interests of Iran than direct Norwegian-Iranian relations.

Sandberg: who is he and what could make him a viable target for intelligence operations?

This is a presentation of Sandberg taken from the web page of the Ministry of Trade, Industry and Fisheries (https://www.regjeringen.no/no/dep/nfd/organisation/y-per-sandberg/id2467677/). Note that his marital status is “married” – but he separated from his wife in May 2018. Sandberg is a vocal figure in Norwegian politics. He has been known to be against immigration and a supporter of strict immigration laws. He has repeatedly been accused of racism, especially by the opposition. He has long held top positions in the Progress Party, which has been a part of a coalition cabinet together with the conservatives (Høyre), and more recently also with the moderately liberalist party “Venstre” (meaning left but it is not a socialist party). Sandberg is known for multiple controversies, summarized on this Wikipedia page: https://en.wikipedia.org/wiki/Per_Sandberg#Controversies. This involves addressing the parliament after having too much to drink, losing his driver’s license due to speeding and finally he was also convicted for violence against an asylum seeker in 1997.

Sandberg has been married since 2010 to 2018 to Line Miriam Sandberg, who has been working as a state secretary for the Ministry of Health since 2017. They recently separated.

His new girlfriend

Sandbergs new girlfriend came to Norway when she was 16 (or 13/14 the first time according to some sources) to flee forced marriage to a 60-year old man in Iran. She is now a Norwegian citizen and is 28 years old. She has participated in several beauty contests in 2013-2014. After she first came to Norway, she was not granted asylum and returned to Iran. Iran sent her back to Norway again because she did not have any identification papers when arriving, and she was adopted by a Norwegian family. A summary of known facts about Letnes and how she gained access to Iran after being returned to Norway without ID papers when she was a teenager was written in Norwegian by Mahmod Fahramand (https://www.nettavisen.no/meninger/farahmand/per-sandbergs-utfordring/3423519653.html). Farahmand is currently a consultant with the auditing and consulting firm BDO and has background from the Norwegian armed forces. He often writes opinion pieces about security related topics. To summarize some of Farahmand’s points.

  • Letnes was returned to Norway and was later adopted by her foster family
  • She has been a “go-to-person” for journalists wanting to get in touch with Iranian officials and has been known to have close relationships with the Iranian embassy in Oslo
  • Iran does not allow Iranian-born indivdiuals to enter Iran without an Iranian passport. If they do not have this, they will need to get access to their birth certificate or otherwise prove to the Iranian government that they in fact have a right to an Iranian passport. Since Letnes fled Iran to seek protection from the threat of her family, it seems she must have gotten access to this without contacting her family, Farahmand argues.

Letnes had her application for asylum turned down 3 times before getting it approve. The reason for the change of the decision of the immigration authorities in 2008 is not known (Norw: https://www.nrk.no/trondelag/–jeg-er-kjempeglad-1.6236578). In addition, it has become known in media in the last few days that Letnes applied for a job with Sandberg’s ministry, suggesting she could act as a translator and guide for Sandberg’s communications with Iran in matters related to fishery and aquaculture trade, which she did not get. Sandberg denied any knowledge of this prior to media inquiring about it. The job application was sent in 2016. She also registered a sole proprietorship in January this year, B & H GENERAL TRADING COMPANY. BAHAREH LETNES, a company to trade with Iran in fish, natural gas, oil and technology (corporate registration information: https://w2.brreg.no/enhet/sok/detalj.jsp?orgnr=920188095). The company has according to Letnes not had any activity so far, according to media reports.

A honeytrap? Possibly. A security breach? For sure.

The arguments from Farahmand’s article above, together with the fact that Letnes tried to get a job for Sandberg in 2016, could easily indicate that Letnes sought to get close to Sandberg. She has sought multiple touchpoints with him since he was appointed Minister of Fisheries in 2015.

This would be a classical honeytrap, although a relatively public one. Sandberg has failed to follow security protocol on many occasions in his dealings with Letnes and Iran. Obvious signs of poor security awareness on behalf of the Minister:

  • He brought his government issued cell phone to Iran and left it unattended for longer periods of time where they stayed at the time
  • He did not tell the office of the Prime Minister about his travel to Iran before leaving. This is a breach of security protocol for Norwegian ministers
  • His separation from his wife became known in May this year
  • He has announced his “original vacation plans got smashed, so the trip to Iran was a last-minute decision”. He was supposed to go on holiday to Turkey, which he had also reported to his Ministry and the office of the Prime Minister, in accordance with security protocol (Norw: https://www.aftenposten.no/norge/politikk/i/e1xzJO/Fiskeriminister-Per-Sandberg-bekrefter-at-han-reiste-til-Iran-uten-a-informere-departementet-eller-statsministeren )
  • The Norwegian government was made aware of Sandberg’s presence in Iran when they received an e-mail from the Iranian embassy in Oslo, requesting official meetings with Minister Sandberg while he was in Iran

Iranian TTP

According to Kjell Grandhagen, former head of Norwegian military intelligence, Iran has a very capable and modern intelligence organization. He holds it as highly likely that Sandberg’s government issued phone, which he left unattended a lot of the time while in Iran, has been hacked (https://www.digi.no/artikler/tidligere-sjef-for-e-tjenesten-tror-per-sandbergs-mobil-har-blitt-hacket/442930). According to this CSO summary, Iran has serious capabilities within both HUMINT and cyber domains. Considering the known cyber capabilities of Iran, and the looming sanctions from the Trump administration, getting both information and leverage over a key politician in a NATO country becomes even more interesting, not only to Iran but also to Russia.

Coming back to Iran’s more recent tighter cooperation with Russia, it is not unlikely that they are also initiating a closer relationship when it comes to intelligence gathering. The use of honey traps has been a long-standing Russian tactic for information gathering and getting leverage over decision makers. In 2015, Norwegian police warned against Russian intelligence operations targeting politicians, including the use of honey traps (https://finance.yahoo.com/news/norwegian-police-warning-citizens-against-195510994.html).

A summary: why is he still in office?

The facts and arguments presented above should indicate two things very clearly:

  • Based on publicly known information, it is clearly possible that Iranian intelligence is targeting Per Sandberg. They may have an asset close to him, as well as having had physical access to his smartphone that has direct access to classified information systems.
  • Further, Sandberg has broken established security protocol, and although admitting this, he does not seem to appreciate the potential impact

The effect of a top leader not taking security seriously is very unfortunate. Good security awareness in an organization depends heavily on the visible actions of its people at the top – in business as well as in politics. A breach of security policy without getting any personal consequences on this level sends a very poor message to other politicians and government officials. It also sends a message to adversaries that targeting top-level politicians is likely to work, even if there are numerous indicators of a security breach. There should be no other possible conclusion of this than relieving Mr. Sandberg of his position – which would set him free to further develop his relationship with the Iranian beauty queen.

Making Django, Elastic Beanstalk and AWS RDS play well together

A couple of days ago I decided I should learn a bit more hands-on AWS stuff. So I created a free tier AWS account, and looked around. I decided I’d take a common use case; deploy a web application to Elastic Beanstalk and add a domain and SSL.

Setting up tools

Step 1: reading documentation. AWS has a lot of documentation, and it is mostly written in a friendly manner with easy-to-follow instructions. Based on the documentation I opted for using the command line Elastic Beanstalk tool. To use this you need Python and pip. You can install it with the command

pip install awsebcli –upgrade

If you are having a permissions problem with doing this, you can throw in a “–user” flag at the end of that command. This will install the tool you need to create and manage EB environments from your command line. Since it is a Python utility it works on Windows, as well as Mac and Linux. Installing this did not pose any hiccups. You can read more about how to set this tool up and updating your system path here: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3-install.html.

Before using it you need to set it up. Issue the command

eb init.

This will give you a prompt asking for a number of things, like region to set up in, etc.

Learning point: if you want to set up a database, such as MySQL in the EB environment, you should use the database option when issuing the next command. Anyway to set up your environment, use

eb create https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb3-create.html

If you want a database in your environment add the –db flag with the desired options; you cannot create the database in the EB Console (web-based interface) afterwards, at least not for micro instances that are allowed in the free tier. According to someone on Stack Overflow, this is a bug in AWS that you can wait for them to fix – or use the command line option (supposedly that works but it is not what I did).

If you create a database in your EB environment, your DB will be terminated too if you terminate that environment. You may not want that, so you can consider setting up an external database and connecting to it outside of EB. That is what I did, and there’s more about that a little further down this post.

Creating a Django app

To have something to deploy I created a Django app. This is an asocial network; you can post things with links and hashtags but you can’t follow other users or anything like that. It has user management and uses the default Django admin system and authentication system (session based). I called it woodscreaming and you can view it here: woodscreaming.com.

Setting up a virtual environment

First, to avoid mixing up things and creating a requirements file that works, create a virtual environment. For this I like to use the tool virtualenv (works on all platforms, can be installed with pip if you don’t have it):

virtualenv –python=python venv

“venv” is the name of your virtual environment. Everything you install when the environment is active will be contained in that environment, and you have all dependencies under control (think of it like a semi-container-solution). To activate the environment on Linux/Mac:

source venv/bin/activate

On Windows:

venv\Script\activate

When you have all the dependencies your app needs in place, run

pip freeze > requrements.txt

This creates a requirements.txt file that EB will use to install your app in the cloud.

Adding EB configuration files to the Django project

To make things work, you also need to add some EB specific configuration files to your Django project. Create a folder named .ebextensions in your project’s root folder. In this folder you will need to add a django.config file with the following contents:

option_settings:
  aws:elasticbeanstalk:container:python:
    WSGIPath: projectname/wsgi.py

Of course you need to change the word projectname into the name of your project. This tells EB where to find your wsgi file. This file describes how the web server should be set up and is a Python standard.

You should also tell EB to run migrations to get your data models to work with your database. Adding a file (I called it db-migrate.config) to the .ebextensions folder fixes this. Here’s what you need to add to that file:

container_commands:
  01_migrate:
    command: "django-admin.py migrate"
    leader_only: true
option_settings:
  aws:elasticbeanstalk:application:environment:
    DJANGO_SETTINGS_MODULE: discproject.settings

You should also create a folder called .elasticbeanstalk. The command line client will populate this with a YAML file called config.yml tha tells EB what resources are needed (you don’t need to edit this file yourself).

That’s it to begin with – some changes need to be made when adding an RDS database and setting up http to https forwarding.

Deploying to EB

Deploying to EB is very easy, you simply deactivate your virtual environment by issuing the command “deactivate” and then you run

eb deploy

It now zips your source, uploads it to AWS and installs it and provisions the resources defined in your config.yml file. It takes a while, and then it is done. Then you can see your web app online by issuing the command

eb open

The app will get its own URL automatically, of the format “something.aws-region.elasticbeanstalk.com”. It does not get an SSL certificate (https) automatically – you will need to set up a custom domain for that (more about that later). Anyway, opening it up shows the web app running in the cloud and I am able to use it.

Dev database vs prod database

By default django-admin.py sets up a project hat uses an SQLite database; a single file SQL database that is popular for persistent storage in mobile apps and embedded applications. When deploying your development environment’s database is deployed too, and with each redeploy you will overwrite it. It is not great for concurrent operations, and obviously overwriting all user data on each deploy is not going to work. There are ways around this if you want to stick to SQLite but that is normally not the best solution for a web app database (although it is great for development).

Next we look at how we can create a database in the cloud and use that with our production environment, while using the SQLite one in local development.

Adding an RDS database

Attempt 1: Using the EB Console

In the EB console (the web interface), if you go to “Configuration”, there is a card for “Database” and an option to “modify”. There you can set up your desired database instance and select apply. The problem is… it doesn’t work for some reason. The deployment fails due to some permission error. I’m sure it is possible to fix but I didn’t bother fiddling enough with it to do that. And as mentioned above; if you terminate the environment you will also terminate the database.

Attempt 2: Setting up and RDS database external to EB

This worked. Basically following AWS documentation on how to set it up was quick and easy:

  • Go to RDS, create a new instance. Select the type of database engine, EC2 instance type etc.
  • Select db name, username, password (remember to write those down – I use secure notes in LastPass for things like this). Set the DB instance to be “public” to allow queries from outside your VPC to reach it.
  • Add the RDS security group to your EB EC2 instance. This is important – if you do not do this, it is not possible to query the database from EB.

To add that security group in EB you need to go to the EB console, head to configuration and then select the card for instances. Select “modify” and then head to the security groups table – add the RDS one (it is automatically generated and named something like rds-default-1) and click apply. Because the database is external to EB you also need to add environment variables for the connection. To do this, head to the console again and select “modify” on the software card. Add the following environment variables:

  • RDS_DB_NAME
  • RDS_HOST_NAME
  • RDS_PORT
  • RDS_USERNAME
  • RDS_PASSWORD

The values are found in your RDS instance overview (head to the RDS console, select your instance, and you find the variables a bit down on the page). Now, you also need to tell your Python app to read and use these. Add this to your Django settings file:

if 'RDS_HOSTNAME' in os.environ:
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'NAME': os.environ['RDS_DB_NAME'],
            'USER': os.environ['RDS_USERNAME'],
            'PASSWORD': os.environ['RDS_PASSWORD'],
            'HOST': os.environ['RDS_HOSTNAME'],
            'PORT': os.environ['RDS_PORT'],
        }
    }
else:
    DATABASES = {
        'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'dbp.sqlite3'),
        }
    }

After doing this the EB environment health was showing as “green” and all good. But my web app did not show up and the log showed database connection errors. The solution to that was: read the docs. You also need to add the RDS default security group (the one that allows inbound connections) to the allowed sources for inbound connections. Details here: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.RDS.html. After doing this – it works!

Adding a Django superuser to the RDS database

You could SSH into your EC2 instance running the Django app and use the manage.py utility; but this kind of beats the point of having a PaaS that supposedly should be able to configure things without SSH-ing into everything.

To add a Django superuser you should thus add a new Django command to your environment. Here’s a good description of how to do that: https://github.com/codingforentrepreneurs/Guides/blob/master/all/elastic_beanstalk_django.md. You can add the command to your db-migrate.config file in the .ebextensions folder.

Configuring DNS with Route 53

Now, having the default URL is no fun, and you can’t add SSL on that one. So we need to set up DNS. I chose to buy a domain name from Amazon and then set up DNS with Route 53. Setting that up for an EB environment is super-easy: you make an A record as alias to your EB environment URL.

Adding an SSL certificate that terminates on the load balancer

Now that we have a working domain name, and we’ve set up the DNS records we need, we can add an SSL certificate. The easiest way to provision the certificate is to use Amazons certificate management service. You provision one for your domain, and you can verify by adding a CNAME record to your DNS hosted zone in Route 53.

The next thing you need to do to make things work is add the certificate to your Elastic Beanstalk environment. Depending on your threat model and your needs, you can choose the simple route of terminating https on the load balancer (good enough for most cases), or you can set up AWS to also use secure protocols in internal traffic (behind the load balancer). I chose to terminate traffic on the load balancer.

The AWS docs explains how to do this by adding a secure listener on the load balancer: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/configuring-https-elb.html.

Forwarding http to https

To forward http traffic to https there are several ways this can be done. The easiest is to set up forwarding on the Apache server. Since we are not using SSH to fiddle with the server directly, we do this by adding a configuration file to our .ebextensions folder in the Django project, and then redeploying. Adding a file https.config with the following contents does the job:

files:
    "/etc/httpd/conf.d/ssl_rewrite.conf":
        mode: "000644"
        owner: root
        group: root
        content: |
            RewriteEngine On
            <If "-n '%{HTTP:X-Forwarded-Proto}' && %{HTTP:X-Forwarded-Proto} != 'https'">
            RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R,L]
            </If>

Summary

This post is a walk-through of getting the essentials done to use Elastic Beanstalk to serve a web application:

  • Create an environment and deploy an app
  • Use config files to manage server processes and configurations
  • Setting up an external RDS database and connect to it using environment variables
  • Configuring a custom domain name and setting up DNS
  • Adding SSL termination on the load balancer
  • Adding a http to https rewrite rule to Apache on the web server using a config file