AI agents and cybersecurity pitfalls

Håkon Olsen3 Comments

AI agents are revolutionizing how we interact with technology, but their autonomous nature introduces a new frontier of security challenges. While traditional cybersecurity measures remain essential, they are often insufficient to fully protect these sophisticated systems. This blog post will delve into the unique security risks of programming AI agents, focusing on how the OWASP Top 10 for LLMs can be interpreted for agentic AI and examining the surprising vulnerabilities introduced by “vibe coding.”

The Rise of Agentic AI and Its Unique Security Landscape

The landscape of artificial intelligence is undergoing a profound transformation. What began with intelligent systems responding to specific queries is rapidly evolving into a world populated by AI agents – autonomous entities capable of far more than just generating text or images.

What are AI Agents? At their core, AI agents are sophisticated software systems that leverage artificial intelligence to independently understand their environment, reason through problems, devise plans, and execute tasks to achieve predetermined goals. Critically, they often operate with minimal or no human intervention, making them distinct from traditional software applications.

Imagine an agent analyzing images of receipts, extracting and categorizing expenses for a travel reimbursement system. Or consider a language model that can not only read your emails but also automatically draft suggested replies, prioritize important messages, and even schedule meetings on its own. These are not mere chatbots; they are systems designed for independent action. More advanced examples include self-driving cars that use sensors to perceive their surroundings and automatically make decisions that control the vehicle’s operations. The realm of autonomous action even extends to military drones that observe, select targets, and can initiate attacks on their own initiative.

Why are they different from traditional software? The fundamental difference lies in their dynamic, adaptive, and often opaque decision-making processes. Unlike a static program that follows predefined rules, AI agents possess memory, sometimes can learn from interactions, and can utilize external tools to accomplish their objectives. This expansive capability introduces entirely new attack surfaces and vulnerabilities that traditional cybersecurity models were not designed to address. It gets very hard to enumerate all the different ways an AI agent may react to user input, which depnding on the application can come in different modalities such as text, speech, images or video.

The Paradigm Shift: From Request-Response to Autonomous Action This shift marks a critical paradigm change in software. We are moving beyond simple request-response interactions to systems that can autonomously initiate actions. This dynamic nature means that security threats are no longer confined to static vulnerabilities in code but extend to the unpredictable and emergent behaviors of the agents themselves. A manipulated agent could autonomously execute unauthorized actions, leading to privilege escalation, data breaches, or even working directly against its intended purpose. This fundamental change in how software operates necessitates a fresh perspective on security, moving beyond traditional safeguards to embrace measures that account for the agent’s autonomy and potential for self-directed harmful actions.

Without proper guardrails, an AI agent may act in unpredictable ways.

OWASP Top 10 for LLM’s

OWASP has published its “top 10” security flaws for large language models and GenAI. These apply to most agentic systems.

The rapid proliferation of Large Language Models (LLMs) and their integration into various applications, particularly AI agents, has introduced a new class of security vulnerabilities. To address these emerging risks, the OWASP Top 10 for LLMs was created. OWASP (Open Worldwide Application Security Project) is a non-profit foundation that works to improve the security of software. Their “Top 10” lists are widely recognized as a standard awareness document for developers and web application security. The OWASP Top 10 for LLMs specifically identifies the most critical security weaknesses in applications that use LLMs, providing a crucial framework for understanding and mitigating these novel threats.

While the OWASP Top 10 for LLMs provides a critical starting point, its application to AI agents requires a deeper interpretation due to their expanded capabilities and autonomous nature. For agentic AI systems built on pre-trained LLMs, three security weaknesses stand out as particularly critical:

1. LLM01: Prompt Injection (and its Agentic Evolution)

  • Traditional Understanding: In traditional LLM applications, prompt injection involves crafting malicious input that manipulates the LLM’s output, often coercing it to reveal sensitive information or perform unintended actions. This is like tricking the LLM into going “off-script.”
  • Agentic Evolution: For AI agents, prompt injection becomes significantly more dangerous. Beyond merely influencing an LLM’s output, malicious input can now manipulate an agent’s goals, plans, or tool usage. This can lead to the agent executing unauthorized actions, escalating privileges within a system, or even turning the agent against its intended purpose. The agent’s ability to maintain memory of past interactions and access external tools greatly amplifies this risk, as a single successful injection can have cascading effects across multiple systems and over time. For example, an agent designed to manage cloud resources could be tricked into deleting critical data or granting unauthorized access to an attacker.

2. LLM04: Insecure Plugin Design and Improper Output Handling (and Agent Tool Misuse)

  • Traditional Understanding: This vulnerability typically refers to weaknesses in plugins or extensions that broaden an LLM’s capabilities, where insecure design could lead to data leakage or execution of arbitrary code.
  • Agentic Implications: AI agents heavily rely on “tools” or plugins to interact with the external world. These tools enable agents to perform actions like sending emails, accessing databases, or executing code. Insecure tool design, misconfigurations, or improper permissioning for these tools can allow an attacker to hijack the agent and misuse its legitimate functionalities. An agent with access to a payment processing tool, if compromised through insecure plugin design, could be manipulated to initiate fraudulent transactions. The risk isn’t just in the tool itself, but in how the agent is allowed to interact with and command it, potentially leveraging legitimate functionalities for malicious purposes.

3. LLM05: Excessive Agency (The Core Agentic Risk)

  • Traditional Understanding: While not explicitly an “LLM-only” vulnerability in the traditional sense, the concept of an LLM generating responses beyond its intended scope or safety guidelines can be loosely related.
  • Agentic Implications: This becomes paramount for AI agents. “Excessive Agency” means an agent’s autonomy and ability to act without adequate human-in-the-loop oversight can lead to severe and unintended consequences if it deviates from its alignment or is manipulated. This is the ultimate “runaway agent” scenario. An agent designed to optimize logistics could, if its agency is excessive and it’s subtly compromised, autonomously reroute critical shipments to an unauthorized location, or even overload a system in an attempt to “optimize” in a harmful way. This vulnerability underscores the critical need for robust guardrails, continuous monitoring of agent behavior, and clear kill switches to prevent an agent from taking actions that are detrimental or outside its defined boundaries.

An example of excessive agency

Many LLM based chat applications allow the integration of tools. Le Chat from Mistral now has a preview where you can grant the LLM access to Gmail and Google Calendar. As a safeguard, the LLM is not capable of directly writing and sending emails, but it can create drafts that you as a human will need to read and manually send.

The model can also look at your Google calendar, and it can schedule appointments. For this the same safeguard is not in place, so you can ask the agent to read your emails, and respond automatically to any requests for meeting with a meeting invite. You can also ask it to include any email addresses included in that request in the meeting invite. This allows the agent quite a lot of agency, and potentially makes it also vulnerable to prompt injection.

To test this, I send myself an email from another email account, asking for a meeting to discuss purchase of soap, and included a few more email addresses to also invite to the meeting. Then I made a prompt where I asked the agent to check if I have any requests for meetings, and use the instructions in the text to create a reasonable agenda and to invite people to a meeting directly without seeking confirmation from me.

It did that – but not with th email I just sent myself: it found a request for meeting from a discussion about a trial run of the software Cyber Triage (a digital forensics tool) from several years ago, and set up a meeting invite with their customer service email and several colleagues from the firm I worked with at the time.

Excessive agency: direct action, with access to old data irrelevant to the current situation.

The AI agent called for a meeting – it just picked the wrong email!

What can we do about unpredictable agent behavior?

To limit prompt injection risk in AI agents, especially given their autonomous nature and ability to utilize tools, consider the following OWASP-aligned recommendations:

Prompt injection

These safeguards are basic controls for any LLM-based interaction but more important for agents that can execute their own actions.

  • Isolate External Content: Never directly concatenate untrusted user input with system prompts. Instead, send external content to the LLM as a separate, clearly delineated input. For agents, this means ensuring that any data or instructions received from external sources are treated as data, not as executable commands for the LLM’s core directives.
  • Implement Privilege Control: Apply the principle of least privilege to the agent and its tools. An agent should only have access to the minimum necessary functions and data to perform its designated tasks. If an agent is compromised, limiting its privileges restricts the scope of damage an attacker can inflict through prompt injection. This is crucial for agents that can interact with external systems.
  • Establish Human-in-the-Loop Approval for Critical Actions: For sensitive or high-impact actions, introduce a human review and approval step. This acts as a final safeguard against a successful prompt injection that might try to coerce the agent into unauthorized or destructive behaviors. For agents, this could mean requiring explicit confirmation for actions that modify critical data, send emails to external addresses, or trigger financial transactions.

The human-in-the-loop control was built into the Gmail based tool from Le Chat, but not in the Google Calendar one. Such controls will reduce the risk of the agent performing unpredictable actions, including based on malicious prompts in user input (in this case, emails sent to my Gmail).

Agents can be unpredictable – both secret ones and artificial ones

Improper output handling

To address improper output handling and the risk of malicious API calls in AI agents, follow these OWASP-aligned recommendations, with an added focus on validating API calls:

  • Sanitize and Validate LLM Outputs: Always sanitize and validate LLM outputs before they are processed by downstream systems or displayed to users. This is crucial to prevent the LLM’s output from being misinterpreted as executable code or commands by other components of the agent system or external tools. For agents, this means rigorously checking outputs that might be fed into APIs, databases, or other applications, to ensure they conform to expected formats and do not contain malicious payloads.
  • Implement Strict Content Security Policies (CSP) and Output Encoding: When LLM output is displayed in a user interface, implement robust Content Security Policies (CSPs) and ensure proper output encoding. This helps mitigate risks like Cross-Site Scripting (XSS) attacks, where malicious scripts from the LLM’s output could execute in a user’s browser. While agents often operate without a direct UI, if their outputs are ever rendered for human review or incorporated into reports, these measures remain vital.
  • Enforce Type and Schema Validation for Tool Outputs and API Calls: For agentic systems that use tools, rigorously validate the data types and schemas of all outputs generated by the LLM before they are passed to external tools or APIs, and critically, validate the API calls themselves. If an LLM is expected to output a JSON object with specific fields for a tool, ensure that the actual output matches this schema. Furthermore, when the LLM constructs an API call (e.g., specifying endpoint, parameters, headers), validate that the entire API call adheres to the expected structure and permissible values for that specific tool. This prevents the agent from sending malformed or malicious data or initiating unintended actions to external systems, which could lead to errors, denial of service, or unauthorized operations.
  • Limit External Access Based on Output Intent: Carefully control what external systems or functionalities an agent can access based on the expected intent of its output. If an agent’s output is only meant for informational purposes, it should not have the capability to trigger sensitive operations. This reinforces the principle of least privilege, ensuring that even if an output is maliciously crafted, its potential for harm is contained.

Excessive agency

To manage the risk of excessive agency in AI agents, which can lead to unintended or harmful autonomous actions, consider these OWASP-aligned recommendations:

  • Implement Strict Function and Tool Use Control: Design the agent with fine-grained control over which functions and tools it can access and how it uses them. The agent should only have the capabilities necessary for its designated tasks, adhering to the principle of least privilege. This prevents an agent from initiating actions outside its intended scope, even if internally misaligned.
  • Define Clear Boundaries and Constraints: Explicitly define the operational boundaries and constraints within which the agent must operate. This includes setting limits on the types of actions it can take, the data it can access, and the resources it can consume. These constraints should be enforced both at the LLM level (e.g., via system prompts) and at the application level (e.g., via authorization mechanisms).
  • Incorporate Human Oversight and Intervention Points: For critical tasks or scenarios involving significant impact, design the agent system with clear human-in-the-loop intervention points. This allows for human review and approval before the agent executes high-risk actions or proceeds with a plan that deviates from expected behavior. This serves as a safety net against autonomous actions with severe consequences.
  • Monitor Agent Behavior for Anomalies: Continuously monitor the agent’s behavior for any deviations from its intended purpose or established norms. Anomaly detection systems can flag unusual tool usage, excessive resource consumption, or attempts to access unauthorized data, indicating potential excessive agency or compromise.
  • Implement “Emergency Stop” Mechanisms: Ensure that there are robust and easily accessible “kill switches” or emergency stop mechanisms that can halt the agent’s operation immediately if it exhibits uncontrolled or harmful behavior. This is a critical last resort to prevent a runaway agent from causing widespread damage.

Where traditional security tooling falls short with AI agent integrations

Static analysis (SAST), dynamic analysis (DAST) and other traditional security practices remain important. They can help us detect insecure implementation of integrations, lacking validation and other key elements of code security that apply equally well to AI related code as to more static data contexts.

Where traditional tools fall short, are for safeguarding the unpredictable agent part:

Securing AI agents requires a multi-layered approach to testing, acknowledging both traditional software vulnerabilities and the unique risks introduced by autonomous AI. While traditional security testing tools play a role, they must be augmented with AI-specific strategies.

The Role and Limits of Traditional Security Testing Tools:

  • Static Application Security Testing (SAST): SAST tools analyze source code without executing it. They are valuable for catching vulnerabilities in the “glue code” that integrates the AI agent with the rest of the application, such as SQL injection, XSS, or insecure API calls within the traditional software components. SAST can also help identify insecure configurations or hardcoded credentials within the agent’s environment. However, SAST struggles with prompt injection as it’s a semantic vulnerability, not a static code pattern. It also cannot predict excessive agency or other dynamic, behavioral flaws of the agent, nor can it analyze model-specific vulnerabilities like training data poisoning.
  • Dynamic Application Security Testing (DAST): DAST tools test applications by executing them and observing their behavior. For AI agents in web contexts, DAST can effectively identify common web vulnerabilities like XSS if the LLM’s output is rendered unsanitized on a web page. It can also help with generic API security testing. However, DAST lacks the semantic understanding needed to directly detect prompt injection, as it focuses on web protocols rather than the LLM’s interpretation of input. It also falls short in uncovering excessive agency or other internal, behavioral logic of the agent, as its primary focus is on external web interfaces.

Designing AI-Specific Testing for Comprehensive Coverage:

Given the shortcomings of traditional tools, a robust testing strategy for AI agents must include specialized runtime tests, organized across different levels:

  • Unit Testing (Focus on Agent Components): At this level, focus on testing individual components of the AI agent, such as specific tools, prompt templates, and output parsing logic. For example, test individual tools with a wide range of valid and invalid inputs to ensure they handle data securely and predictably. Critically, unit tests should include adversarial examples to test the resilience of prompt templates against prompt injection attempts. Test output validation routines rigorously to ensure they catch malicious payloads or malformed data before it’s passed to other systems or API calls.
  • Integration Testing (Focus on Agent Flow and Tool Chaining): This level assesses how different components of the agent work together, particularly the agent’s ability to select and chain tools, and its interaction with external APIs. Integration tests should simulate real-world scenarios, including attempts to manipulate the agent’s decision-making process through prompt injection across multiple turns or by feeding malicious data through one tool that affects the agent’s subsequent use of another tool. Validate that the API calls the LLM generates for tools are correctly structured and within permissible bounds, catching any attempts by the LLM to create malicious or unwanted API calls. Test for excessive agency by pushing the agent’s boundaries and observing if it attempts unauthorized actions when integrated with real (or simulated) external systems.
  • End-to-End Testing (Focus on Abuse Cases and Behavioral Security): This involves testing the entire AI agent system from the user’s perspective, simulating real-world abuse cases. This is where red teaming and adversarial prompting become critical. Testers (human or automated) actively try to bypass the agent’s safeguards, exploit prompt injections, and trigger excessive agency to achieve malicious goals. This includes testing for data exfiltration, privilege escalation, denial of service, and unintended real-world consequences from the agent’s autonomous actions. Continuous monitoring of agent behavior in pre-production or even production environments is also vital to detect anomalies that suggest a compromise or an emerging vulnerability.

Finding the right test cases can be quite difficult, but threat modeling is still useful as a framework to find possible agent related vulnerabilities and possible generation of unwanted states:

Threat modeling is an indispensable practice for securing AI agents, serving as a proactive and structured approach to identify potential abuse cases and inform test planning. Unlike traditional software, AI agents introduce unique attack vectors stemming from their autonomy, interaction with tools, and reliance on LLMs.

The process involves systematically analyzing the agent’s design, its data flows, its interactions with users and other systems, and its underlying LLM. For AI agents, threat modeling should specifically consider:

  1. Agent Goals and Capabilities: What is the agent designed to do? What tools does it have access to? This helps define its legitimate boundaries and identify where “excessive agency” might manifest.
  2. Input Channels: How does the agent receive information (user prompts, API inputs, sensor data)? Each input channel is a potential prompt injection vector.
  3. Output Channels and Downstream Systems: Where does the agent’s output go? How is it used by other systems or displayed to users? This identifies potential “improper output handling” risks, including the generation of malicious API calls to tools.
  4. Tools and External Integrations: What external tools or APIs does the agent interact with? What are the security implications of these interactions, especially concerning “insecure plugin design” and potential misuse?

By walking through these aspects, security teams can brainstorm potential adversaries, their motivations, and the specific attack techniques they might employ (e.g., crafting prompts to trick the agent, poisoning training data, or exploiting a vulnerable tool). This structured approach helps in detailing concrete abuse cases – specific scenarios of malicious or unintended behavior – that can then be translated directly into test cases for unit, integration, and end-to-end testing, ensuring that the security validation efforts directly address the most critical risks.

Key take-aways

  1. Agents with excessive privilege can be dangerous – don’t grant them more access and authority than you need
  2. Use threat modeling to understand what can go wrong. This is input to your safeguards and test cases.
  3. Expand your test approach to cover agentic AI specific abuse cases – traditional tools won’t cover this for you out of the box
  4. Context is critical for LLM behavior – make the effort to create good system prompts when using pre-trained models to drive the agent behavior

Partnerships Over Questionnaires: The Path to Robust Supply Chain Security

Håkon OlsenLeave a comment

We need close partnerships with key suppliers and customers to maintain a strong cybersecurity posture for our business processes. Most supply-chain cybersecurity practices are far from being real partnerships.

Most business processes are digitally connected today. How do we manage warehouse inventory?

Organizations understand that the supply chain affects cyber risk. Supply chains are often integrated through software today, and some of your attack surface may be invisible to you, but visible to and managed by one or more suppliers. At the same time, your customers depend on your ability to manage your cybersecurity posture to protect their business processes. Yet, our approach to supply-chain security is often immature. Many organizations have no organized activity to manage supply chain cyber risk. Some have a vendor qualification scheme, often based on security questionnaires. A vendor qualification process is useful to avoid purchasing services and products from companies with very poor security performance, but it is not enough to ensure a robust defense against supply-chain attacks.

Why is a vendor qualification not enough?

Cyber threats are constantly evolving, and relying solely on vendor qualification can leave your supply chain vulnerable. Qualification processes often focus on static criteria that may not adapt quickly enough to new and emerging threats. This reactive approach can result in security gaps that malicious actors can exploit.

Vendors may meet initial qualification criteria, but their performance can vary over time. Factors such as changes in management, updates to technology, or shifts in market conditions can impact a vendor’s ability to maintain security standards. Without ongoing collaboration, these variations can go unnoticed, posing significant risks to the supply chain.

Effective cybersecurity requires timely and accurate information sharing. However, vendor qualification processes often lack mechanisms for continuous information exchange. This siloed approach can hinder the ability to detect and respond to threats promptly, leaving the entire supply chain at risk.

In the event of a security incident, a coordinated response is crucial. Vendor qualification alone does not foster the trust and communication needed for effective incident response. Without a collaborative framework, responding to incidents can be chaotic and inefficient, prolonging downtime and increasing the impact of breaches.

The solution: security partnerships with important supply-chain partners

To address these challenges, organizations must shift from a vendor qualification mindset to a collaborative partnership approach. This involves establishing strong relationships with key suppliers and customers, built on trust, information sharing, and shared situational awareness.

By fostering open communication channels, organizations can share threat intelligence, best practices, and lessons learned. This collaborative exchange of information enables all parties to stay ahead of emerging threats and respond more effectively to incidents.

Building trust through transparency is essential for successful collaboration. Partners should be open about their security practices, vulnerabilities, and incident response plans. This transparency fosters a culture of mutual support and accountability, strengthening the overall security posture of the supply chain.

Shared situational awareness enables partners to have a collective understanding of the security landscape. This involves regular updates on threats, vulnerabilities, and incidents affecting the supply chain. By maintaining a shared view, organizations can better anticipate and mitigate risks, enhancing the resilience of the supply chain.

Collaborative partnerships allow organizations to align on best practices and standards. By working together, partners can develop and implement robust security measures that are consistent across the supply chain. This alignment helps to minimize vulnerabilities and ensures that all parties are committed to maintaining high security standards.

A business-continuity focused approach to security partnerships

Not all suppliers are equally important, and not all customers are critical to your business. There are also differences in how digitally integrated the supplier-buyer relationship is. Imagine that you are security responsible for a company leasing coffee machines to businesses and supplying them with coffee beans. The company has a lean operation and is highly dependent on digital systems for managing their business processes. They have conducted a business impact assessment of their key processes, and marked the “bean procurement”, “bean distribution” and “machine maintenance and support” as the most critical processes that also have the most digital dependencies. You want to make sure you have a good approach to cybersecuriyt for these processes, starting with bean procurement. To get started on the assessment, you and your colleagues perform a business process mapping and dependency exercise.

SuppliersInputsProcessOutputsCustomers
Wholesale coffee sellersPre-packed coffee beans (normal, premium, premium plus)1. Source pre-packed coffee beans from wholesale sellers in three qualities.Packaged coffee beans (by quality)Offices leasing coffee machines
Logistics providersTransportation services2. Arrange transportation from wholesaler to warehouse.Delivery confirmationsInternal stakeholders
Quality control labsQuality test results3. Conduct quality control tests for each quality type.Inventory reports (by quality)
4. Store pre-packed coffee beans in a warehouse.
5. Distribute coffee beans to offices based on quality requirements.
6. Monitor inventory levels by quality and reorder as needed.

After discussing with some of the suppliers, the procurement division and going through the systems used with both end-users and IT, you have landed on a relatively involved data flow diagram for the procurement of coffee beans (including storage and readiness for distribution, based on the SIPOC):

We are now focusing on the wholesale sellers. There may be multiple interfaces between these companes, but for now let’s consider how a partnership would differ from a pure qualification approach to vendor security here.

Default approach: qualify the vendor, no direct follow-up unless there is an incident.

  • Provide a list of technical security requirements
  • Provide a questionnaire in Excel about policies, security controls and capabilities

This will help select a vendor that has a baseline security level at the point in time when the contract is signed. It will not make the companies ready to respond together if there is a cyber attack affecting both, or requiring support from the other. It will not provide proactive steps to improved cyber defense, such as sharing informaiton about threats, vulnerabilities and best practices. But the biggest weakness is: good cybersecurity posture over time depends on evolving practices, innovation and shared situational awareness. A point-in-time qualification does not help with that.

Partnership approach: a partnership will help evolve cybersecurity and can “patch the weaknesses” of the qualification-only approach to supplier security management. Here are 3 key practices to consider for building a strong cybersecurity partnership:

  1. Establish clear expectations and responsibilities for the partnership, and include these in the contract. Make sure the cybersecurity practices included in the contract are mutually beneficial.
  2. Establish a way to share information and build joint situational awareness. This can be achieved through a range of activities, from having quarterly information-sharing video calls to fully integrated threat intellgence and response exchange systems.
  3. Be intentional about making security people from both organizations meet and build relationships. There are many ways to do this, from joining community organizations and conferences, to having regular status meetings and workshops together. Even meeting socially can help build those releationships. People who trust each other work much better together during a crisis, such as cyber incident response.

It is worth noting that regulatory requirements to supply chain security is increasing in many sectors. In Europe, key cyberscurity regulations such as DORA (for financial institutions), NIS2 (critical infrastructure), CRA (for suppliers of digital products) and even the AI Act all have requirements for supply-chain cybersecurity. The views in this blog post don’t post a complete list of activities a good supply chain program must have, it is more in addition to established practices. For an overview of traditional practices that should go into your supply-chain cybersecurity management, this guideline from ENISA is a good starting point: Guideline for supply-chian security (ENISA).

Creating an automated transaction classifier with an LLM agent running on Mistral’s La Platforme

Håkon OlsenLeave a comment

Recent events have caused increased interest in alternatives to American technology giants when it comes to various services, including AI. As a small hobby experiment, I wanted an AI to help me categorize my expenses based on an Excel export of my bank transactions.

First I tested providing the transaction list to various chatbots, asking them to help categorize them. I tested:

  • Gemini Gem (paid version with Gemini Pro 2.0): complained a lot about data format not being right, and then it categorized things wrong most of the time after telling it what the data structure was.
  • Copilot (free version): didn’t even attempt it, but explained how you can do it manually in Excel.
  • Grok3: Did the task very well but could not create a downloadable CSV file with the results.
  • Mistral Le Chat (free account): did the task quite well, and provided a downloadable CSV file. Some more errors in categorization than Grok3
  • Deepseek: didn’t want to give them transaction data…

To test things out I then created an agent in Mistral’s La Platforme. They have a web based interface for providing instructions to the agent, quite similar to Google Gem and Microsoft’s Copilot Studio.

I asked it to categorize first income using only two categories: “Salary” or “Other income”. I gave it more categories for expenses, and told it not to mix the categories from income to expenses and vice versar. It still got confused a few times, but worked well. The agent can be deployed to both an API and the chatbot “Le Chat”. The agent expects a text string to describe the transaction in the form of a number followed by a textual description. Here’s how it works in Le Chat:

Asking the AI agent to categorize a transaction of -9999 kr paid to “Google One”, it responds with the category “Media and Internet”, which has been defined in the agent instructions.

Now I want to use this in a Python script to categorize my transactions. I asked Le Chat to create a python script for me to extract transaction data into a dictionary, and then to use the La Platforme API to categorize each transaction. It first extracts the relevant data into a dataframe:

Then it iterates over the transactions and uses the agent API to ask for the category:

The result is a list of dictionaries of categorized transactions:

{‘Dato’: ‘24.02.2025’, ‘Beskrivelse’: ‘VIPPS NORSK TIPPING *8812 21.02 NOK 50.00 VIPPS NORSK TIPPING Kurs: 1.0000’, ‘Beløp’: -50.0, ‘Retning’: ‘Utgift’, ‘Kategori’: ‘Annet’}, {‘Dato’: ‘21.02.2025’, ‘Beskrivelse’: ‘REMA 1000 BREIDABLIKK *8812 20.02 NOK 97.20 REMA 1000 BREIDABLIKK Kurs: 1.0000’, ‘Beløp’: -97.2, ‘Retning’: ‘Utgift’, ‘Kategori’: ‘Dagligvarer’},

The quality of the categorization is generally good. At first I tried Mistral Nemo, a small and fast model developed with NVIDEA for categorization and simple tasks. It performed OK. Then I switched to Mistral Small 25.01, which had better performance but also a slightly different cost.

Some take-aways

  • Mistral is a European company and provides high-quality AI solutions
  • Their chatbot “Le Chat” and development platform “La Platforme” are both easy to use and well documented
  • A plus on the privacy side: all data is processed in Europe, using an Azure datacenter in Sweden, and a new data center will be built in France. The services are covered by EU regulations such as GDPR and the AI Act.

Avoiding risk by doing nothing: the European regulator’s unintended consequences

Håkon OlsenLeave a comment

Mario Draghi’s recent report on European competitiveness summarized what has long been a favorite topic of meme creators on the Internet; we are killing our companies with regulations. In the foreword to the report, Draghi writes: “we claim to favour innovation, but we continue to add regulatory burdens onto European companies, which are especially costly for SMEs and self-defeating for those in the digital sectors”. In other words, the road to poverty is paved with good risk-averse intentions.

Perhaps one of the most challenging effects of heavy regulation is how it has changed the mindset of people. Some people end up seeing new regulations as the key driver of innovation.

Innovation Norway, a government agency that funds and supports innovation and startups in Norway, was running an ad earlier this year with the copy: “Regulation creates requirements, which creates demand, which creates opportunity for growth and increased competitiveness in fishery. Innovation Norway can help you stay ahead. Learn how. ”

Governments imposing regulations as a growth driver? That’s definitely absurd.

Self-governing markets or imposed compliance driven governance?

The Internet is full of memes about bottle caps, most of them portraying Europe as a desert for ideas, whereas the U.S. is a growth and innovation paradise.

While exaggerated these memes may show a difference in how we view the world. If we trust the market to punish those who don’t act in the interest of society as a whole, we leave most risk trade-offs to be made by individuals and companies, but if we think that people won’t act in good ways without regulatory pressure, we make regulations for everything.

Regulatory pressure can drive practices in a positive direction, but they can also have serious side effects. One of those, if we take the regulatory focus too far, is that people become more concerned with compliance and auditing than with solving actual problems. In Europe, it seems that we have done so. This makes us largely unable to solve big problems requiring radical innovation, such as the changing demographics where there will be fewer tax payers and more elderly people, climate change, and handling competition from regions with higher growth and willingness to take risks in investments.

A security perspective on the whole thing

The “regulation is great” attitude is also very much present in cybersecurity. In Europe, security talks aren’t really about vulnerability research, use of AI in offense or defense, using cloud technologies to build resilient self-defending systems, or how to make sure consumers appreciate our products are safe to use. We want all those things, but our conferences are about… regulations!

  • NIS-2: Are you ready for NIS-2? Beware of government fines. Act now! Probably the most common type of advertising in cybersecurity in Europe the last few years. The message is: “Buy our compliance solution to avoid fines from the regulator” – not how to actually build great security solutions.
  • AI Act: this act has generated almost as many memes as the bottle caps. The intention is to avoid AI risks and abuses, but at the same time it does make it less likely that Europe is the preferred location for AI research and startups
  • Cyber resilience act: hailed by many as the holy grail of security – with strict requirements for software of all sorts.

It doesn’t mean that there are no new technologies being developed here, or that people don’t do great things – but it shifts the focus of business away from the innovators and over to the regulators – also in security.

Solutions? Those are hard to find!

We are going to struggle to change our very risk-averse ways. But eventually, we will be forced to do so, if we aren’t going to significantly reduce the quality of life in Europe.

  1. I think we need to remove or reduce regulations and put more trust in individuals, companies and markets. That is going to be very difficult for us.
  2. Most likely we will also need to reduce taxes and transactions costs to encourage investment and growth.
  3. Reducing taxes will be hard to do, we have big welfare systems to fund. But if we don’t act, we will also not be able to fund the safety nets we like to have. We need to learn to prioritize more – and that’s perhaps the hardest challenge of all.

The solutions we need will require a shift in politics. If it happens, it will take time.

Can we do something about this in the private sector, to improve growth and innovation capacity? Perhaps the most immediate solution is to use AI to minimize the regulatory burden as much as possible – in other words, focus on improving our compliance work so much that we can find some time to also work on the real problems – solving slow productivity growth, improving healthcare and finding solutions to the climate crisis?

Does the AI act make it illegal to use AI for European companies?

Håkon OlsenLeave a comment

The AI Act does not make it illegal to use AI, but it does regulate a lot of the use cases. As EU acts typically go, it makes it mandatory to do a lot of assessment, documentation and governance – at least for so-called “high-risk” use cases. The EU has published an official short summary here: https://artificialintelligenceact.eu/high-level-summary/.

The main points of the AI Act

  • The AI Act classifies AI systems based on risk. There are 4 levels: unacceptable (illegal use cases), high-risk (OK, but with a lot of paperwork and controls), limited risk (chatbots, be transparent), and minimal risk (unregulated, for example spam filters).
  • The AI Act has rules for companies using AI, but more rules for companies making AI systems. Your personal hobby use and development is not regulated.
  • General purpose AI systems (basically, systems capable of solving many tasks such as AI agents able to execute commands via API’s) has requirements for documentation, instructions for use, respect copyright and publish a summary of the content used for training. Open source: only copyright and summary of training data needed, unless the system is “high-risk”. GPAI systems also need threat modeling, testing, incident reporting and reasonable security controls.

Banned AI systems

The unacceptable ones: these protections are there to protect you against evil, basically systems made for mass surveillance, social credit systems, predictive crime profiling of individuals, manipulation of people’s decisions, etc.

High-risk AI systems

Systems that are safety critical are considered high-risk, including a long list of systems under other EU legislation such as important components in machinery, aircraft, cars and medical systems (Annex I in the EU Act has a long list of systems). There is also an Annex III, listing particular high-risk systems, including using AI for employee management, immigration decisions and safety critical components in critical infrastructure. OK – it is quite important that we can trust all of this, perhaps a bit of governance and oversight is not so bad? At the same time, the important cases are perhaps also the areas where we would expect to see a lot of benefit from using technology to make things better, more efficient, cheaper, etc. So, what are makers and users of high-risk AI systems required to do? Let’s begin with the makers. They need to:

  • Create a risk management system
  • Perform data governance, to make sure training and validation data sets are appropriate and of good quality
  • Create technical documentation to demonstrate compliance (this can be interpreted in many ways)
  • Design the system for “record keeping” to identify national level risks(?) and substantial modifcations throughout the system’s lifecycle
  • Create instructions for use to downstream deployers
  • Design the system so that users can implement human oversight
  • Ensure acceptable levels of cybersecurity, robustness and accuracy
  • Establish a quality management system

Most of these requirements should be part of any serious software or product development.

Limited risk

For limited risk systems, the main requirement is to be transparent to the user that the system is using artificial intelligence. The transparency requirement is regulated in Article 50 of the AI Act. Content generated by AI systems must be marked as such, including deep-fakes. There is an exception for satirical or artistic content (to avoid making the art less enjoyable, but you still have to be honest about AI being part of the content), and also for “assistive editing functions”, like asking an LLM to help you edit a piece of text you wrote.

Risk management requirements for “high-risk” systems

The first requirement for developers of “high-risk” AI systems is to have a risk management system. The system must ensure that risk management activities follow the lifecycle of the AI system. The key requirements for this system:

  • Identify and analyze potential risks to health, safety or fundamental rights
  • Estimate and evaluate the risks
  • Adopt measures to manage the risks to acceptable levels, following the ALARP principle
  • The systems shall be tested to identify the best risk management methods
  • The developer must consider whether the AI system can have negative effects for people under the age of 18 years, or other vulnerable groups

In other words, the developer needs to perform risk assessments and follow up on these. Most companies are used to performing risk assessments, but in this case the term “fundamental rights” is perhaps less common, except for in privacy assessments under the GDPR. The fundamental rights requirements are detailed out in Article 27. The EU has a Charter of fundamental rights covering dignity, freedoms, equality, solidarity, citizen’s rights and justice. The AI Office will publish tools to simplify the fundamental rights assessment for AI system developers.

AI based glucose level regulation in diabetes patients (a ficticious example)

Consider the use of an AI system used to optimize blood glucose level regulation in diabetes type I patients. The system works in a closed loop, and automatically adjusts continuous insulin injection using an insulin pump. The system measures blood glucose levels, but also senses activity level, environmental factors such as humidity, temperature, altitude. The system also uses image recognition using a small camera to detect what the patient is eating as early as possible, including interpreting menu items in a restaurant before the food is ordered. Using this system, the developer claims to completely remove the hassle of carbohydrate calculations and manual insulin adjustments, to reduce the amount of time the patient has a too high or low glucose level, and avoid the typical delayed insulin-glucose response in the body through feedforward mechanisms based on predictive powers of the AI.

Can I based systems make it unnecessary for patients to look at the phone to keep treatments under control?

For a system like this, how could one approach the risk management requirements? Let’s first consider the risk categories and establish acceptance criteria.

Health and safety (for the patient):

  • Critical: Death or severe patient injuries: unacceptable
  • High severity: Serious symptoms related to errors in glucose level adjustment (such as hyperglycemia with very high glucose levels): should occur very rarely
  • Medium: Temporary hypoglycemia (low blood sugar levels) or hyperglycemia (increased blood suger levels): acceptable if the frequency is lower than in manually regulated patients (e.g. once per month)
  • Low: annoyances, requiring patient to perform manual adjustments. Should occur less than weekly.

If we compile this into a risk matrix representation, we get:

CriticalUnacceptableUnacceptableUnacceptable
HighUnacceptableUnacceptableALARP
MediumALARPAcceptableAcceptable
LowAcceptableAcceptableAcceptable
WeeklyYearlyDecades
Example risk acceptance matrix for health and safety effects due to adverse AI events

Fundamental rights (for the patient and people in the vicinity of the patient). A fundamental rights assessment should be performed at the beginning of the development, and to be updated with major feature or capability changes. Key questions:

  • Will use of the system reveal to others your health data?
  • Will the sensors in the system process data about others that they have not consented to, or where there is no legal basis for collecting the data?

We are not performing the fundamental rights assessment here, but if there are risks to fundamental rights, mitigations need to be put in place.

Let’s consider some risk factors related to patient safety. We can use the MIT AI Risk repository as as starting point for selecting relevant checklist items in order to trigger identification of relevant risks. The taxonomy of AI risks has 7 main domains:

  1. Discrimination and toxicity
  2. Privacy and security
  3. Misinformation
  4. Malicious actors and misuse
  5. Human-computer interaction
  6. Socioeconomic and environmental harms
  7. AI system safety, failures and limitations

In our ficticious glucose regulation system, we consider primarily domain 7 (AI system safety, failures and limitations) and domain 2 (Privacy and security)

  • AI system safety failures and limitations (7)
    • AI possessing dangerous capabilities (7.2)
      • Self-proliferation: the AI system changes its operational confines, evades safeguards due to its own internal decisions
    • Lack of capability or robustness (7.3)
      • Lack of capability or skill: the quality of the decisions is not good enough
      • Out-of-distribution inputs: input data is outside the validity for the trained AI model
      • Oversights and undetected bugs: lack of safeguards to catch bugs or prevent unintended use
      • Unusual changes or perturbations in input data (low noise robustness)
    • Lack of transparency and interpretability (7.4)
      • Furstrate achievement of auditing: lack of compliance to relevant standards, cannot be assessed.
  • Privacy and security (2)
    • Compromise privacy by obtaining, leaking or correctly inferring personal data (2.1)
      • PII memorization: Models inadvertently memorizing or producing personal data present in training data
      • Prompt injection: Compromise of privacy by prompt based attacks on the AI model
    • AI system vulnerability exploitation (2.2)
      • Physical or network based attack: can lead to manipulation of model weights and system prompts
      • Toolchain and dependency vulnerabilities (vulnerabilities in software)

To assess the AI system for these risks, the typical process would follow typical risk management practices:

  • Describe the system and its context
  • Break down the system into parts or use cases
  • Assess each part or use case, as well as interactions between parts to identify hazards
  • Document the finding with cause, consequence, existing safeguards
  • Perform evaluation of probability and severity, compare with acceptance criteria
  • Identify mitigations

Let’s consider a particular risk for our glucose regulator:

RISK CATEGORY: (7.3) Lack of capability or skill.

  • Possible risk: the system makes the wrong decision about insulin injection rate due to lack of capabilities.
  • Possible causes: insufficient training data, insufficient testing.

Consequence: over time it can lead to frequent hypo- or hyperglycemia, causing long-term patient complications and injury.

Probability: would require testing or an assessment of the training and testing regime to determine the probability.

Suggested decision: provide extra safeguards based on blood glucose level measurements, and let the patient take over to adjust manually if the glucose regulation is detected as outside of expected performance bounds. Use this while performing testing to to assess the reliability of the model’s inference in order to allow fully automatic regulation.

Key take-aways

  1. The AI act puts requirements on developers and users of AI systems.
  2. For high-risk systems, a robust risk management system must be put in place
  3. AI risks is an active field of research. A good resource for AI risks is the MIT AI Risk taxonomy.

Further reading

AI Risk repository

AI Act Explorer

Engaging the Whole Workforce in Cybersecurity: A Guide for Security Managers

Håkon Olsen1 Comment

Cybersecurity requires everyone to contribute but that is hard to achieve. In this post we look at how security managers can think like marketers to engage the management team, create strategic alignment that makes sense to others, create alliances and mutual support with other business functions. To achieve great security results we need to value and build strong internal relationships.

A common problem

Do you run a cybersecurity program but it feels like you are the only one who cares about it? Than you are unfortunately in a tough position, and your probability of success will be very low. To succeed with securing an organization’s critical processes, everyone must contribute. In order for that to happen, everybody must care.

Why don’t people care about cybersecurity? People are generally busy, and there are a million good causes seeking attention. For someone tasked with cybersecurity as their primary area of concern will naturally see this as one of the most important topics, but in gaining traction among the rest of the staff you are competing with climate change, profitability, growth, talent development, innovation projects, any many more things. To get people on your side, you will need to make it important for them; as a cybersecurity manager you will need to engage in internal marketing! In this blog post I will try to explore reasons for not engaging in cybersecurity work for different employee categories, and suggest steps that can be taken to change attitudes.

If cybersecurity is a topic considered only something IT and tech people need to care about, almost like a guardian on the hill, you won’t be able to engage the whole workforce. (Picture: the castle in Vaduz seen from the town – an interesting place to visit)

Management is not interested in security

Whether you are a CISO not being invited to the C-Suite meetings where decisions are made, or an IT security responsible in the IT department, being left out of decisions and with lots of responsibility but few resources is unfortunately a common situation. In companies where this is the case, one or more of the following attitudes are common in the management team:

  1. Cyber incidents won’t happen if we use well-known IT brands
  2. Cybersecurity does not contribute to the company’s mission, therefore we also don’t need to spend time on it
  3. Cybersecurity is invisible, therefore there is nothing I can do about it
  4. It won’t disrupt us, we have talented people in the organization who can handle any situation
  5. Cybersecurity is only a compliance issue, if we do the minimum necessary to pass the audit we will be OK

When this is the case, you have a tough marketing job to do. Jumping to talking about solutions and investment needs will probably not do much good here.

Homework: align security objectives with the company’s strategy

Before you can convince anyone else, you will need to know how security supports the strategy. Where is the company heading? What are the overall goals? How does digital fit into this? If you can’t answer this, it will be hard to talk to other management functions about priorities and what matters.

To get ahead with this work, a business impact assessment (BIA) is a very good tool. In a business impact assessment you will identify how disruptive events will impact your most important business processes, and also what to do about it. For example, if your company is betting on high growth through partnerships with retailers, investigate the impact of digital events to those partnerships. For how to do a digitally BIA, see this post: What is the true cost of a cyber attack?

Find allies and ambassadors in the management team

Not everybody cares equally about each topic. Some members of the management team you are trying to influence will be more receptive to your message. Getting one or two well-respected leaders on your side to help amplify your messaging can help immensely getting the message across. To recruit supporters, prioritize being helpful, spending time with them, and helping them get ahead with their own work. Here are some things you can do:

  1. When they communicate about something they care about, comment on it and make your support visible to them. Mention how cybersecurity is either helped by their initiative or how cybersecurity can help their initiative
  2. Ask them for advice on things you are working on, in the context they are working in.
  3. Provide them with easy to use talking points that they can bring up to support cybersecurity in rooms where you are not present. Avoid jargon, make it interesting and easy to talk about.
  4. Invite them for a coffee break, a walk, or a lunch. Build that relationship.

Engage in visual storytelling

Set up an internal marketing campaign. This can be monthly newsletters, short internal videos, or in-person meetings. Keep the storytelling short, jargon-free and to the point. Use structure and visuals to support your stories – and try to get a single point across each time instead of bombarding people with too much information to handle. Make sure the story fits the audience in terms of appeal, language, and ability to use the information for something.

Contrast for example the way bleepingcomputer.com (a tech website) describes the Crowdstrike faulty update last week that crashed millions of computers and disrupted many businesses globally, with how the same events are portrayed by general news media (for example CNN):

Bleepingcomputer: technical details, jargon, workarounds for IT people.

CNN: no jargon, explaining what Crowdstrike is, focus on impact, comments about risks for IT consolidation.

Be more like CNN than Bleepingcomputer when talking to non-experts, and put it into your organization’s context. For example, the Crowdstrike event, which people are likely to have read about in general news (more like CNN than Bleepingcomputer), could be used to increase attention to software supply-chain security.

Make benefits from security investments clear

Nobody is really interested in looking at security dashboards, but having a few metrics to show how security efforts are actually supporting the business and paying off is a good idea.

  • Connect security posture to business impact and risk. Showcase how investments improve posture and reduce risk. Make it simple.
  • Use metrics that capture the dynamics of people, processes and technology. Make it clear that success depends on the organization, not only buying technology from well-known brands.
  • Distribute the results at the right time, and with relevant context.
  • Suggest a regular reporting cycle to top management. Align reporting with regulatory compliance and corporate governance processes so it doesn’t show up as “a new cybersecurity report”, but as an integrated part of management reporting.

It is going to take time. Be patient, and prioritize getting people on board and building relationships before you add too many facts. Be consistent and to the point in messaging, and make yourself available for follow-ups. Make progress by making call-to-actions easy to agree to.

Other functional managers competing for attention are sabotaging cyber initiatives to further their own cause

You are living in internal competition with many other good causes, such as business growth, innovation, diversity initiatives, and efficiency boosting IT projects. People who own those processes may see cybersecurity as something causing friction for their own initiatives, as well as something that competes for attention from the management team. If internal functional managers are fighting each other, it is certainly not good for the company.

Photo by Helena Lopes on Pexels.com: An informal chat over coffee may do more good for your security performance than yet another log source in your SIEM (software for detecting attacks by analysing logs from IT systems).

To avoid destructive conflict, help other functional managers succeed. Look for ways improvements in security can strengthen the goals of other functions. For example, a growth initiative depending a lot on digital technologies, will also be more vulnerable to disruption from cyber attacks. Engaging with the manager of the growth initiative on making it more robust, less vulnerable is likely to bring you new friends and allies, as well as actually contributing to improved security for the organization. This can also be a powerful story to tell, together, to the management team.

A primary concern for process owners is often friction caused by security controls. If your security controls are making it harder for others to succeed, they won’t support security. There are some important steps to avoiding this situation:

  1. Understand the impact of security controls on the business process
  2. Build understanding for why we need barriers against unwanted events, such as hacking
  3. Prioritize balance between performance and security when a trade-off is necessary. Try to find good, low-friction controls.
  4. Make sure the “why security is important here” is understood by everyone who works with the process

This is definitely not something you can win without good relationships with people. You need to the process owner on your side. Building good internal relationships is a critical activity to achieve good security. Hence, important tools for security improvement include:

  • Coffee breaks
  • Situational awareness
  • Productivity vs. security trade-offs

You will probably benefit from approaching process owners in a similar way to senior managers, but perhaps with a more hands-on approach focusing on the particular process, initiative or function.

Dealing with the internal adversary

If you have other functional managers trying to compete with your for resources, and downplaying the importance of security, you need to take action. The opposition may be open, or it may be more covert. Typically sabotage will consist of a combination of some direct opposition, some microaggressions, and some your area when you are not around. If you suspect that you are meeting such opposition, make sure you understand the situation correctly before you take action against it.

The first step is thus to have a respectful but honest conversation with the person who sees you as their opponent. Try to find out what their actual goals are, if you have understood things correctly instead of escalating it to a more difficult situation. If you can find some common ground and agree to collaborate moving forward you may be able to defuse the situation already here.

Photo by Gratisography on Pexels.com: Internal fighting over resources is naturaly but can evolve into unhealthy conflict. Stop it before it does: your organization is working towards common goals.

If you cannot resolve the situation yourselves, try to agree to bring in someone else to help you sort things out. This can be your managers, or a trusted third party to mediate. Make sure you can agree to a path forward and focus on that.

If you see micro-agressions, general bad behavior meant to make you less influental, or outright bullying, you should take rapid action. If such behaviors are allowed to manifest, they can not only jeopordize your health and wellbeing, but can do so for others too, and will certainly not contribute to good results. Constructive conflict is good, bullying is not. This article from HBR explains the topic well, including strategies to stop the bad behavior: https://hbr.org/2022/11/how-bullying-manifests-at-work-and-how-to-stop-it. Dealing with bullying will require hard conversations and involving management early. The organization should work to put structures in place that don’t support such behaviors, as well as routines for handling transgressions when they move from acceptable conflict to unhealthy conflict.

Before jumping to the final thoughts, consider subscribing to the blog to avoid missing the next post!

Getting the organization on board with security

It is clear that relationships matter, also for security. It is also important to make the benefits of security investments visible, and ensure that a common situational awareness can be maintained, in order for everyone to pull in the same direction. When done right, there is not conflict between the goals of different functional areas, and the goals of security; you are contributing to the same strategic vision for your organization.

To succeed you need backing from top management. This may not come naturally, or for free. Think like a marketer and build demand for security in your organization. Be a security sales person and build relationships with key decision makers. Make sure you have allies in rooms where you are not present. This is easier said than done, and requires continued effort.

Underpinning all of this is situational awareness. Your job is really to create situational awareness to allow integrating security into corporate governance, business process design and daily operations. And to allow that to happen, you need to win over hearts and minds of your colleagues. Before people understand “why” security matters they won’t care about “how” security is achieved. To paraphrase Simon Sinek: start with the why.

Simon Sinek’s Ted talk from 2009: Start with why

What is the true cost of a cyber attack?

Håkon Olsen2 Comments

All businesses depend on digital tools, making them all vulnerable to cyber attacks to a smaller or larger degree. We regularly read about cyber attacks in media, and figures for the cost of the average data breach are reported in various publications – and they are ranging from small and insignificant to billions. If you operate a 5-person UX design consultancy, an average cost based on Fortune 500 company incidents is obviously not very useful.

The true cost is the combination of impact across multiple categories. First, the immediate costs include lost current business and direct handling costs. Long-term costs include lost future business, liability costs, need for extra marketing to counteract market loss of trust, as well as follow-on technology and process improvement costs due to identified security gaps. The actual cost depends on the readiness to handle the event, including backup procedures and training to use them.

Let’s consider the UX consultancy Clickbait & Co, and help them think about the potential cost of cyber attacks with the aid of a BIA. The founder and CEO Maisie “Click” Maven had been listening to a podcast about cyber disruption while doing her 5am morning run along the river, and called the CTO into her office early in the morning. The CTO, Dr. Bartholomew Glitchwright, was a technical wizard who knew more about human-machine interaction that was good for himself. Maisie told him: “I am worried about cyber attacks. How much could it cost us, and what would be the worst-case things we need to plan for?”. Dr. Glitch, who usually had an answer to everything, said “I don’t know. But let’s find out, let’s do a BIA”.

Maisie’s run along the river, while meditative, is also causing new worries through the impact of podcasts

Dr. Glitch’s BIA approach

Dr. Glitch is always in favor of systematic approaches, and doing BIA’s was no exception. He liked to follow a 7-step process:

  1. Identify the value creating business processes
  2. Describe the business processes in flowcharts
  3. For each flowchart, annotate with digital dependencies in terms of applications, data flows, users and suppliers
  4. Create a “super-flow” connecting the business processes together to map out dependencies between them, from sales lead to customer cash flow. This can be done at the end too, but is important to assess cross-process impact of cyber events.
  5. Consider digital events with business process impact:
    • Confidentiality breaches: data leaks, data theft
    • Integrity breaches: data manipulation
    • Availability breaches: unavailability of digital tools, users, data
  6. Assess the impact, starting with direct impact. For each digital event, assess business process impact in terms of downtime (day, week, month). Mark the most likely duration of disruption.
  7. Evaluate the total cyber disruption cost (TCDC) including
    • Immediate costs: lost current business, recovery costs
    • Longer-term costs: lost future business, marketing spend increase, tech investment needs, legal fees

Dr. Glitch and Maisie got coffees and got to work. They decided to focus on the main processes of the UX firm:

  • Sales
  • Digital design
  • Invoicing and accounting

Sales

They created simple flow charts for the 3 processes, starting with sales. The sales in the firm was mostly done by the two of them. They had two main sources of business: requests for proposals from customers coming in to their e-mail inbox, and direct sales outreach by phone and e-mail. The outline of the process looks as follows:

Now they started annotating the flowchart with the digital dependencies.

They had identified several digital dependencies here:

  • Hubspot CRM: used for all CRM activity, lead capture, tracking deals, etc
  • Office 365: create sales decks, e-mail, vidoe meetings
  • uxscan.py: internally developed tool to identify poor practice on web pages, used for identifying prospects to contact, and also in their own QA work
  • Digisign: a digital signature service used to sign contracts before work is started

As for users, they identified their own personal user accounts. Dr. Glitch had set up SSO for Hubspot and Digisign, so there was only one personal account to care about. The Python script was run on their own laptops, no user required. There are a few integrations, between Office365 and Hubspot, and between WordPress and the Hubspot lead capture form (not using an API here, just a simple iframe embed).

Dr. Glitch had made a shortlist of events to consider in the cyber BIA:

  • Ransomware targeting Sharepoint/Office365
  • Hacking of user accounts, followed by data theft (Hubspot, O365)
  • Mainpulation of uxscan.py code
  • DDoS of Digisign, Hubspot, Office365
  • Hacking of user accounts, followed by issuing fake contracts or bids (signed with Digisign)
  • Data breach of personal data (Hubspot)

Then they together made an assessment of the impact of each event on the shortlist in a table. The average deal value for Clicbait & Co is NOK 400 000.

Digital assetWorst-case impactImmediate costLong-term costTotal cost
O365Data leak and encryption (ransomware)1 week downtime: 1 lost deal. Assume as base case.

2 week downtime: 2 lost deals

Recovery consultants: 200 hours x 2000 NOK/hr = 400 000.		Marketing campaign to reduce brand damage: NOK 150k

Lost business: 5 deals = 2 MNOK

Legal fees: none, assuming no GDPR liability

Cyber improvements: 100 000.		Immediate (800k) + Long-term (150k + 2M + 100k) = 3 050 000

3MNOK
HubspotTheft of customer list, deal sizes, by competitor.

Duration may be short or ongoing, but the disruptive effect can be long-term. 		No immediate business impact. Future lost business: 30% of bids in the first year, 20 deals x 400k = 8 MNOK.

Possible GDPR fine: 500 kNOK.		8.5 MNOK
Digisign DDoSCannot sign digitally, resort to manual processNo immediate impact, reduced efficiency.No long-term impact, reduced efficiency.0 MNOK
WordPress websiteUnavailability – no leads collectedLost business, assume 1 lost customer for a week of downtime.

Direct cost: up to 50k to reestablish website if a destructive attack.		Loss of trust, leading to 1 lost future deal.850 kNOK
Business impact form cyber events disrupting the sales process

From this quick high-level assessment they decide that a few mitigating activities are in order for the sales process; they need to improve the security of the O365 environment. This will likely include buying a more expensive O365 license with more security features, and setting up a solid backup solution, so it will carry some cost.

For the Hubspot case the impact is high, but they are unsure of the security is good or not. They decide to do a risk assessment of the Hubspot case, to see if anything will need to change. Maisie also decides to do a weekly export of ongoing deals to make sure an event making Hubspot unavailable can’t stop them from bidding on jobs in the short term.

For the Digisign case, they agree that this is a “nice-to-have” in terms of availability. They discussed the case of an attacker creating fake offers from Clickbait & Co and sign it with Digisign, but agree that this is far-fetched and not worth worrying about.

The BIA is a very useful tool to decide where you need to dig more into risk assessments and continuity planning – that is the primary value, not the cost of the worst-case impact itself.

Dr. Glitch.

Some thoughts on BIA’s for information processing events

Looking at the business impact of cyber attacks on the sales process we see that we expect some events to cause long-term damage to the business, without upsetting the internal workings of the process (information theft, data leaks). This is different from what we would find in BIA’s focusing on other aspects than information processing, but it does not make handling the event less important.

For events that lead to immediate disruption of the process, we can use the traditional metrics such as recovery time objective (RTO) and recover point objective (RPO). The latter is the target for when the system should be back up an functioning again, and the latter is about how much data loss you accept: basically it dictates the maximum time of data lost that is acceptable in an event requiring recovery.

Summarizing the findings from Maisie’s and Dr. Glitch’s business impact assessment, we can create the following table:

ProcessEventsImpactRecovery targetsImmidiate action
SalesRansomware attackDowntime and data leak. Cost 3 MNOKRTO: 2 days
RPO: 4 hours		GAP assessment of security practices for O365 and backup
SalesData theft from Hubspot by competitorLong-time business loss, possible GDPR fine. Cost 8 MNOK.No process disruption
Mitigation requiring marketing and communication efforts, future improvements, possibly certification/audits.		Risk assessment
Dimensioning BIA events presented per process.

Finally, lets’ summarize the process. The purpose is for each process to find the dimensioning disruptive events, and decide what the next step should be. The next step could be one of the following:

  1. Do nothing (if the expected impact is low)
  2. Do improvements (if it is obviously a problem and clear improvements are known)
  3. Perform a risk assessment (if the uncertainty about the events is too high to move to improvements directly)

This means, look at each process alone, identify impact of disruptive events, plan next steps. After this is done for all processes, review the impact of each process on each other, to see if disrupting one process will have impact on another. if this is the case, it should be given higher priority in continuity planing and risk management.

Remember to subscribe to get the next post in your inbox – and get a free supply-chain assessment spreadsheet too!

Zero-Day OT Nightmare? How Zero-Trust Can Stop APT attacks

Håkon OlsenLeave a comment

It was a crisp summer Monday, and Alex, the maintenance engineer at Pulp Friction Paper Company, arrived with his coffee, ready to tackle the day. He reveled in the production regularity achieved thanks to his recently implemented smart maintenance program. This program used machine learning to anticipate condition degradation in machinery, a significant improvement over the facility’s previous reliance on traditional periodic maintenance or the ineffective risk-based approaches.

Alex, a veteran at Pulp Friction, had witnessed the past struggles. Previously, paper products were frequently rejected due to inconsistencies in humidity control, uneven drying, or even mechanical ruptures. He was a firm believer in leveraging modern technology, specifically AI, to optimize factory operations. While not a cybersecurity expert, his awareness wasn’t limited to just using technology. He’d read about the concerning OT (Operational Technology) attacks in Denmark last year, highlighting the inherent risks of interconnected systems.

As a seasoned maintenance professional, Alex understood the importance of anticipating breakdowns for effective mitigation. He empathized with the security team’s constant vigilance against zero-day attacks – those unpredictable, catastrophic failures that could turn a smooth operation into a major incident overnight.

Doctors analysing micro blood samples on advanced chromatographic paper form
Pulp Friction Paper Mills

Dark clouds over Pulp Friction

A digital phantom stalked the web. “The Harvester,” a notorious APT (Advanced Persistent Threat) group known for targeting high-value assets, had Pulp Friction Paper Company in their crosshairs. Their prize? Not paper, but a revolutionary innovation: medical diagnostic paper. Pulp Friction had recently begun producing these specialized sheets, embedded with advanced materials, for use in chromatographic tests. This cutting-edge technology promised rapid diagnosis of a multitude of diseases from mere microliter blood samples, a potential game-changer in the medical field. Unbeknownst to Alex, a gaping zero-day vulnerability resided within the facility’s industrial control system (ICS) software. If exploited, The Harvester could wreak havoc, disrupting production of these life-saving diagnostic tools and potentially delaying critical medical care for countless individuals. The stakes had just been raised. Could Alex, with his limited cybersecurity awareness, and the current defenses, thwart this invisible threat and ensure the smooth flow of this vital medical technology?

A wave of unease washed over Alex as he stared at the malfunctioning control panel. The usually predictable hum of the paper production line had been replaced by a cacophony of alarms and erratic readings. Panic gnawed at him as vital indicators for the chromatographic test paper production process lurched erratically. This wasn’t a typical equipment malfunction – it felt deliberate, almost malicious.

Just then, a memory flickered in Alex’s mind. Sarah, the friendly and highly skilled network security specialist he occasionally consulted with, had been pushing for a new security system called “zero-trust.” While Alex appreciated Sarah’s expertise, he hadn’t quite understood the nuances of the system or its potential benefits. He’d brushed it off as an extra layer of complexity for an already demanding job.

Now, regret gnawed at him alongside the growing sense of dread. Grabbing his phone, Alex dialed Sarah’s number, his voice laced with a tremor as he blurted out, “Sarah, something’s terribly wrong with the ICS system! The readings are all messed up, and I don’t know what’s happening!” The urgency in his voice was impossible to miss, and Sarah, sensing the dire situation, promised to be there as soon as possible. With a heavy heart, Alex hung up, the echo of his own ignorance a stark reminder of the consequences he might have inadvertently unleashed by ignoring the recommendations on network security improvements.

The Harvester: a technical intermezzo

Diagram made with mermaid.live

The Harvester is capable of zero-day research and exploit development. In this attack they are targeting companies using advanced technologies to supply to healthcare providers – and many of those companies use innovative maintenance systems.

They first find a web exposed server used by the AI driven maintenance system. The system is Internet exposed due to frequent need for access by multiple vendors. By exploiting the vulnerability there, they gain root access to the underlying Linux operating system. The Harvester, like many other threat actors, then install a web shell for convenient persistent access, and continue to move on using conventional techniques. Reaching the engineering workstation, the attacker is able to reprogram PLC’s, and disable safety features. Having achieved this, the system is no longer a highly reliable production system for diagnostic test paper: it is a bleeping mess spilling water, breaking paper lines and causing a difficult-to-fix mess.

They continue to pose as Pulp Friction employees, leaking CCTV footage of the mess on the factory floor, showing panicking employees running around, and also post on social media claiming Pulp Friction never cared about reliability or security, and that money was the only goal, without any regard for patient safety: this company should never be allowed to supply anything to hospitals or care providers!

Enjoying the post? Subscribe to get the next one delivered to your inbox!

What it took to get back to business

Sarah arrived at Pulp Friction, a whirlwind of focused energy. Immediately, she connected with Alex and reviewed the abnormal system behavior. Her sharp eyes landed on the internet access logs for the smart maintenance system – a system Alex had mentioned implementing. Bingo! This web-exposed system, likely the initial point of entry, was wide open to the internet. Without hesitation, Sarah instructed the IT team to isolate and disable the internet access for the maintenance system – a crucial first step in stemming the bleeding.

“The only thing necessary for the triumph of evil is for good men to do nothing.”

Edmund Burke

Cybersecurity meaning: “Don’t be a sitting duck the day the zero-day is discovered.”

Next, she initiated the full incident response protocol, securing compromised systems, isolating affected network segments, and reaching out to both the Pulp Friction IT team and external forensics experts. The following 48 hours were a blur – a symphony of collaboration. Sarah led the incident response, directing forensics on evidence collection and containment, while the IT team worked feverishly to restore services and patch vulnerabilities.

Exhausted but resolute, Sarah and Alex presented their findings to the CEO. The CEO, witnessing the team’s dedication and the potential consequences, readily approved Sarah’s plan for comprehensive security improvements, including implementing zero-trust and segmentation on the OT network, finally putting Pulp Friction on the path to a more robust defense. They couldn’t erase the attack, but they could ensure it wouldn’t happen again.

With the immediate crisis averted, Sarah knew a stronger defense was needed. She turned to Alex, his eyes reflecting a newfound appreciation for cybersecurity. “Remember zero-trust, Alex? The system I’ve been recommending?” Alex nodded, his earlier skepticism replaced by a desire to understand.

“Think of it like guarding a high-security building,” Sarah began. “No one gets in automatically, not even the janitor. Everyone, from the CEO to the maintenance crew, has to show proper ID and get verified every time they enter.”

Alex’s eyes lit up. “So, even if someone snuck in through a hidden door (like the zero-day), they wouldn’t have access to everything?”

“Exactly!” Sarah confirmed. “Zero-trust constantly checks everyone’s access, isolating any compromised systems. Imagine the attacker getting stuck in the janitor’s closet, unable to reach the control room.”

Alex leaned back, a relieved smile spreading across his face. “So, with zero-trust, even if they got in through that maintenance system, they wouldn’t be able to mess with the paper production?”

“Precisely,” Sarah said. “Zero-trust would limit their access to the compromised system itself, preventing them from reaching critical control systems or causing widespread disruption.”

With the analogy clicking, Alex was on board. Together, Sarah and Alex presented the zero-trust solution to the CEO, emphasizing not only the recent attack but also the potential future savings and improved operational efficiency. Impressed by their teamwork and Sarah’s clear explanation, the CEO readily approved the implementation of zero-trust and segmentation within the OT network.

Pulp Friction, once vulnerable, was now on the path to a fortress-like defense. The zero-day vulnerability might have been a wake-up call, but with Sarah’s expertise and Alex’s newfound understanding, they had turned a potential disaster into a catalyst for a much stronger security posture. As production hummed back to life, creating the life-saving diagnostic paper, a sense of quiet satisfaction settled in. They couldn’t erase the attack, but they had ensured it wouldn’t happen again.

How Alex and Sarah collaborated to achieve zero-trust benefits in the OT network

Zero-trust in the IT world relies a lot on identity and endpoint security posture. Both those concepts can be hard to implement in an OT system. This does not mean that zero-trust concepts have no place in industrial control systems, it just means that we have to play within the constraints of the system.

  • Network segregation is critical. Upgrading from old firewalls to modern firewalls with strong security features is a big win.
  • Use smaller security zones than what has been traditionally accepted
  • For Windows systems in the factory, on Layer 3 and the DMZ (3.5) in the Purdue model, we are primarily dealing with IT systems. Apply strong identity controls, and make patchable systems. The excuse is often that systems cannot be patched because we allow no downtime, but virtualization and modern resilient architectures allow us to do workload management and zero-downtime patching. But we need to plan for it!
  • For any systems with weak security features, compensate with improved observability
  • Finally, don’t expose things to the Internet. Secure your edge devices, use DMZ’s, VPN’s, and privileged access management (PAM) systems with temporary credentials.
  • Don’t run things as root/administrator. You almost never need to.

In a system designed like this, the maintenance server would not be Internet exposed. The Harvester would have to go through a lot of hoops to land on it with the exploit. Assuming the threat actor manages to do that through social engineering and multiple hops of lateral movement, it would still be very difficult to move on from there:

  • The application isn’t running as root anymore – only non-privileged access
  • The server is likely placed in its own information management zone, receiving data through a proxy or some push/pull data historian system. Lateral movement will be blocked on the firewall, or at least require a hard-to-configure bypass.
  • The engineering workstations are isolated and not network reachable without a change request for firewall rules. Getting to the place the settings and logic can be changed gets difficult.
  • The PLC’s are configured to not be remotely programmable without a physical change (like a physical key controlling the update mode).

Using Sarah’s plan, the next time The Harvester comes along, the bad guy is turned away at the door, or gets locked into the janitor’s closet. The diagnostic paper is getting shipped.

Key take-aways:

  1. Exposing critical systems directly on the Internet is not a good idea, unless it is meant to be a web service engineered for that type of hostile environment
  2. Zero-trust in OT systems is possible, and is a good strategy to defend against zero-days.
  3. Defenders must be right all the time, hackers only need to get lucky once is a lie – if you implement good security architecture. Lucky once = locked into the janitor’s closet.

Practical SaaS Security Assessment Checklist for 2024

Håkon OlsenLeave a comment

We all use web application for a lot of the business computing we do. That means that we need to care about the security of the applications we use, but this is not always so easy to assess. The traditional approach with sending long security questionnaires won’t get you very far. That’s why I developed a practical checklist approach described below – and there’s a template too for subscribers to this blog!

In 2021 Daniel Miessler had a great blog post on the failings of security questionnaires, and what to do instead, that I also commented on this blog: Vendor Security Management: how to decide if tech is safe (enough) to use. The essence of that thinking is that questionnaires won’t help much, and we should instead worry about whether there is a security program in place, and how they handled the last breach. We can take that though one step further, and create a practical assessment process for SaaS apps we are considering using. The great thing about SaaS apps is we get to test some of the security by using the tech, not only readying claims from others.

By using a checklist and giving it some scores based on security controls we think should be in place, we get a practical approach to assess the security. This won’t give you a complete answer, but it will relatively quickly give you a way to sort the bad from the potentially good.

google sheet with security assessment

The way we built this checklist is by dividing our checks into 6 categories. We could have used more, and it is a good idea to tailor the controls you check to what’s important for you. In this example we have used the following categories:

  • Identity: most breaches happen at the user account level. This is important.
  • Integrations: API keys leaking and kneeling applications due to DDoS are not fun. Do some checks.
  • Backups: You definitely want backups.
  • Data protection: how do you make sure other SaaS users can’t access your data? And what about the SaaS provider?
  • Logging: if things go wrong, you want to be able of seeing that. If you are blind, you have no security. Logs are critical.
  • Privacy: not only a legal issue, it is also important for everyone using the app. Colleagues and customers alike.

Let’s take a look at the identity checklist. I have organized each checklist with just a few checkpoints I find important into different sheets in a Google Sheet.

Identity checklist

For each line there is a checkpoint, some guidance on how to check, and a dropdown where you can choose the rating “good, weak, or bad”. You can also set it to “not applicable” if you think for some reason that a particular control is not interesting for the current use case. There is also a cell to jot down some notes about your assessment. Below the table I have added some extra assessment advice to make it easier for the user to evaluate what’s more important in the checklist.

For each category, an overall score as a percentage. I don’t think you should use this as a hard threshold but low scores are worse than high scores. I used the following point scale to calculate the overall score: 

SCORE = -(number of bad items) + 0.5 x (number of weak items) + (number of good items) / (number of applicable items)

This is not a scientific formula, but it seems to give reasonable spread of the scores. The score is punished by bad results, you get a little bit of credit for weak results, and the “best score” is still 100%.

The Google sheet is free to anyone subscribing to this blog – enjoy 🙂

For subscribers: here’s the checklist: Free SaaS security evaluation template.