Application security in Django projects

This is a quick blog post on how to remove some typical vulnerabilities in Django projects.

coffee shop interior
Even a coffee shop visitor registration app needs to take app security into account

The key aspects we are looking at are:

  • Threat modeling: thinking through what attackers could do
  • Secrets management with dotenv
  • Writing unit tests based on a threat model
  • Checking your dependencies with safety
  • Running static analysis with bandit

Threat model

The app we will use as an example here is a visitor registration app to help restaurants and bars with COVID-19 tracing. The app has the following key users:

  • SaaS administrator: access to funn administration of the app for multiple customers (restaurants and bars)
  • Location administrator: access to visitor lists for individual location
  • Personal user: register visits at participating locations, view their own visit history and control privacy settings
  • Unregistered user: register visits at participating locations, persistent browser session lasting 14 days

The source code for this app is available here: https://github.com/hakdo/besokslogg/.

We use the keywords of the STRIDE method to come up with quick attack scenarios and testable controls. Note that most of these controls will only be testable with custom tests for the application logic.

Attack typeScenarioTestable controls
SpoofingAttacker guesses passwordPassword strength requirement by Django (OK – framework code)

Lockout after 10 wrong consecutive attempts (need to implement in own code) (UNIT)
Tampering
RepudiationAttacker can claim not to have downloaded CSV file of all user visits.CSV export generates log that is not readable with the application user (UNIT)
Information disclosureAttacker abusing lack of access control to gain access to visitor list

Attacker steals cookie with MitM attack

Attacker steals cookie in XSS attack
Test that visitor lists cannot be accessed from view without being logged in as the dedicated service account. (UNIT)

Test that cookies are set with secure flag. (UNIT OR STATIC)

Test that cookies are set with HTTPOnly flag. (UNIT or STATIC)

Test that there are no unsafe injections in templates (STATIC)
Denial of serviceAttacker finds a parameter injection that crashes the applicationCheck that invalid parameters lead to a handled exception (cookies, form inputs, url parameters)
Elevation of privilegeAttacker gains SaaS administrator access through phishing.Check that SaaS administrator login requires a safe MFA pattern (UNIT or MANUAL)
Simple threat model for contact tracing app

Secrets management

Django projects get a lot of their settings from a settings.py file. This file includes sensitive information by default, such as a SECRET_KEY used to generate session cookies or sign web tokens, email configurations and so on. Obviously we don’t want to leak this information. Using python-dotenv is a practical way to deal with this. This package allows you to include a .env file with your secrets as environment variables, and then to include then into settings using os.getenv(‘name_of_variable’). This way the settings.py file will not contain any secrets. Remember to add your .env file to .gitignore to avoid pushing it to a repository. In addition, you should use different values for your development and production environment of all secrets.

from dotenv import load_dotenv
load_dotenv()

SECRET_KEY = os.environ.get('SECRET_KEY')

In the code snippet above, we see that SECRET_KEY is no longer exposed. Use the same technique for email server configuration and other sensitive data.

When deploying to production you need to set the environment variables in that environment using a suitable and secure manner to do it. You should avoid storing configurations in files on the server.

Unit tests

As we saw in the threat model, the typical way to fix a security issue is very similar to the typical way you would fix a bug.

  1. Identify the problem
  2. Identify a control that solves the problem
  3. Define a test case
  4. Implement the test and develop the control

In the visitor registration app, an issue we want to avoid is leaking visitor lists for a location. A control that avoids this is an authorisation check in the view that shows the visitor list. Here’s that code.

@login_required()
def visitorlist(request):
    alertmsg = ''
    try:
        thislocation = Location.objects.filter(service_account = request.user)[0]
        if thislocation:
            visits = Visit.objects.filter(location = thislocation).order_by('-arrival')
            chkdate = request.GET.get("chkdate", "")
            if chkdate:
                mydate = datetime.datetime.strptime(chkdate, "%Y-%m-%d")
                endtime = mydate + datetime.timedelta(days=1)
                visits = Visit.objects.filter(location = thislocation, arrival__gte=mydate, arrival__lte=endtime).order_by('-arrival')
                alertmsg = "Viser besøkende for " + mydate.strftime("%d.%m.%Y")
            return render(request, 'visitor/visitorlist.html', {'visits': visits, 'alertmsg': alertmsg})
    except:
        print('Visitor list failed - wrong service account or no service account')
        return redirect('logout')

Here we see that we first require the user to be logged in to visit this view, and then on line 5 we check to see if we have a location where the currently logged in user is registered as a service account. A service account in this app is what we called a “location administrator” in our role descriptions in the beginning of our blog post. It seems our code already implements the required security controls, but to prove that and to make sure we detect it if someone changes that code, we need to write a unit test.

We have written a test where we have 3 users created in the test suite.

class VisitorListAuthorizationTest(TestCase):

    def setUp(self):
        # Create three users
        user1= User.objects.create(username="user1", password="donkeykong2016")
        user2= User.objects.create(username="user2", password="donkeykong2017")
        user3= User.objects.create(username="user3", password="donkeykong2018")
        user1.save()
        user2.save()

        # Create two locations with assigned service accounts
        location1 = Location.objects.create(service_account=user1)
        location2 = Location.objects.create(service_account=user2)
        location1.save()
        location2.save()
    
    def test_return_code_for_user3_on_visitorlist_is_301(self):
        # Authenticate as user 3
        self.client.login(username='user3', password='donkeykong2018')
        response = self.client.get('/visitorlist/')
        self.assertTrue(response.status_code == 301)
    
    def test_redirect_url_for_user3_on_visitorlist_is_login(self):
        # Authenticate as user 3
        self.client.login(username='user3', password='donkeykong2018')
        response = self.client.get('/visitorlist/', follow=True)
        self.assertRedirects(response, '/login/?next=/visitorlist/', 301)
    
    def test_http_response_is_200_on_user1_get_visitorlist(self):
        self.client.login(username='user1', password='donkeykong2016')
        response = self.client.get('/visitorlist/', follow=True)
        self.assertEqual(response.status_code, 200)

Here we are testing that user3 (which is not assigned as “service account” for any location) will be redirected when visiting the /visitorlist/ url.

We are also testing that the security functionality does not break the user story success for the authorized user, user1, who is assigned as service account for location1.

Here we have checked that the wrong user cannot access the URL without getting redirected, and that it works for the allowed user. If someone changes the logic so that the ownership check is skipped, this test will break. If on the other hand, someone changes the URL configuration so that /visitorlist/ no longer points to this view, the test may or may not break. So being careful about changing the inputs required in tests is important.

Vulnerable open source libraries

According to companies selling scanner solutions for open source libraries, it is one of the most common security problems that people are using vulnerable and outdated libraries. It is definitely easy to get vulnerabilities this way, and as dependencies can be hard to trace manually, having a tool to do so is good. For Python the package safety is a good open source alternative to commercial tools. It is based on the NVD (National Vulnerability Database) from NIST. The database is run by pyup.io and is updated every month (free), or you can pay to get updates faster. If you have a high-stakes app it may pay off to go with a commercial option or to write your own dependency checker.

Running it is as easy as

safety check -r requirements.txt

This will check the dependencies in requirements.txt for known vulnerabilities and give a simple output in the terminal. It can be built into CI/CD pipelines too, as it can export vulnerabilities in multiple formats and also give exit status that can be used in automation.

Static analysis with bandit

Static analysis can check for known anti-patterns in code. A popular choice for looking for vulnerabilities in Python code is bandit. It will test for hardcoded passwords, weak crypto and many other things. It will not catch business logic flaws or architectural bad choices but it is a good help for avoiding pitfalls. Make sure you avoid scanning your virtual environment and tests, unless you want a very long report. Scanning your current project is simple:

bandit -r .

To avoid scanning certain paths, create a .bandit file with defined excludes:

[bandit]
exclude: ./venv/,./*/tests.py

This file will exclude the virtual environment in /venv and all files called “tests.py” in all subfolders of the project directory.

A false positive

Bandit doesn’t know the context of the methods and patterns you use. One of the rules it has is to check if you are using the module random in your code. This is a module is a standard Python modules but it is not cryptographically secure. In other words, creating hashing functions or generating certificates based on random numbers generated by it is a bad idea as crypto analysis could create realistic attacks on the products of such generators. Using the random module for non-security purposes on the other hand, is convenient and unproblematic. Our visitor log app does this, and then bandit tells us we did something naughty:

Test results:
        Issue: [B311:blacklist] Standard pseudo-random generators are not suitable for security/cryptographic purposes.
        Severity: Low   Confidence: High
        Location: ./visitor/views.py:59
        More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b311-random
     58          alco = ''
     59          valcode =  ''.join(random.choice(string.ascii_lowercase) for i in range(6)) 
     60          errmsg = ''    
 

What we see here is the check for cryptographically insecure random numbers. We are using it just to verify that a user knows what they are doing; deleting their own account. The app generates a 6-letter random code that the user has to repeat in text box to delete their account and all associated data. This is not security critical. We can then add a comment # nosec to the line in question, and the scanner will not report on this error again.

Things we will not catch

A static analyser will give you false positives, and there will be dangerous patterns it does not have tests for. There will be things in our threat model we have overlooked, and therefore missing security controls and test requirements. Open source libraries can have vulnerabilities that are not yet in the database used by our scanner tool. Such libraries can also in themselves be malicious by design, or because they have been compromised, and our checks will not catch that. Perhaps fuzzing could, but not always. In spite of this, simple tools like writing a few unit tests, and running some scanners, can remove a lot of weaknesses in an application. Building these into a solid CI/CD pipeline will take you a long way towards “secure by default”.

Deploying Django to app engine with Python 3.7 runtime – fails because it can’t find pip?

Update 1 June 2020: Google tells me the problem is now fixed. I haven’t verified it yet, will update if I find it is not fixed at the next crossroads. Bug tracker link: https://issuetracker.google.com/issues/131663964.

Update 30 March 2020: The problem is still here. Use pip version 19.2.3 now to make it work. Google: can you please go fix this now?

Update 30 April 2019: Problem is back. This time it tries to upgrade to pip 19.1, but the app engine instance is stuck on 19.0.3. Adding pip==19.0.3 in the requirements.txt file saves the deployment.

Update 1 April 2019: now the deploy fails with the same message as described in this post, when the PIP version is specified in the requirements.txt file. Removing the specific pip version line from the requirements file fixes this. I have not seen any change notice or similar from Google on this.

I had an interesting error that took quite some time to hunt down today. These are basically some notes on what caused it and how I tracked it down. I have an app that is deployed to Google App Engine standard. It is running Django and using the Python 3.7 runtime – and it was working quite well for some time. Then yesterday I was going to deploy an update (actually just adding some CSS tweaks), it failed, with a cryptic error message. Running “gcloud app deploy” lead to the following error message:

File upload done.
Updating service [default]…failed.
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build a33ff087-0f47-4f18-8654-********* status: FAILURE.
Error ID: B212CE0B.
Error type: InternalError.
Error message: pip_install_from_wheels had stderr output:
/env/bin/python3.7: No module named pip
error: pip_install_from_wheels returned code: 1.

This is weird: this is a normal Python project using a requirements.txt file for its dependencies. The file was generated using pip freeze, and should not contain anything weird (it doesn’t). Searching the wisdom of the internet reveals that nobody else seems to have this problem, and it only occurred since yesterday. My hunch was that they’ve changed something on the GAE environment that broke something. Searching the net gives us these options:

  • The requirements.txt file has weird encoding and contains Chinese signs/letters? That was not it.
  • This is because you need to install some special packages for using Python3.. was also not the case and would have been weird changing since a few days ago…
  • You need to manually install pip to make it work – which may be the case sometimes but without SSH access to the instance this isn’t obvious how to do.
The trick is often looking at the right logs….

So, being clueless I turned to looking for the right logs to figure out what is going on. Not being an expert on the GAE environment led to some hunting in the web console until I found “Cloud build”, which sounded promising. That was the right place to be – GAE uses a build process in the cloud to first build the application, and then a Docker image to push to the internal Google Cloud Docker repository. Hunting the build log for this finds this little piece of gold:

Step #1 - "builder": INFO pip_install_from_wheels took 0 seconds
Step #1 - "builder": INFO starting: pip_install_from_wheels
Step #1 - "builder": INFO pip_install_from_wheels /env/bin/python3.7 -m pip install --no-deps --prefix /tmp/tmp9Y3aD7/env /tmp/tmppuvw4s/wheel/pip-19.0.1-py2.py3-none-any.whl --disable-pip-version-check
Step #1 - "builder": INFO `pip_install_from_wheels` stdout:
Step #1 - "builder": Processing /tmp/tmppuvw4s/wheel/pip-19.0.1-py2.py3-none-any.whl
Step #1 - "builder": Installing collected packages: pip
Step #1 - "builder": Found existing installation: pip 18.1
Step #1 - "builder": Uninstalling pip-18.1:
Step #1 - "builder": Successfully uninstalled pip-18.1
Step #1 - "builder": Successfully installed pip-19.0.1
Step #1 - "builder": Failed. No module /env/python/pip

Before the weird error we see it is trying to uninstall pip-18.1, then install pip-19.0.1 (a more recent version), and then it can’t find pip afterwards and the build process fails. This has not been configured by me and is probably Google configuring automatic upgrades of some packages during build – and here it breaks the workflow.

Fixing it

The temporary fix was simple. Adding “pip==18.1” to the requirements.txt file, allowed the build process to run and it deployed nicely.

What did we learn from this?

  • API tools give only partial error messages, making debugging hard.
  • Automated upgrade configs are good but can cause things to break in unforeseen ways.
  • Finding the right logs is the key to fixing weird problems