The other day I listened to the Microsoft Threat Intelligence Podcast on my way to work (Spotify link). There they mentioned that the threat actor “Peach Sandstorm” had used Azure Arc as a persistence vector, but not giving much detail about how this would work. This rather unusual technique can be a bit hard to spot, especially on Linux hosts, unless you know where to look. The log file /var/opt/azcmagent/log/hidms.log is a good place to start (if it exists and you are not using Azure Arc, you may be in trouble).

I got a bit curious, so I decided to how that would work, and what kind of logs it would generate on the host (without any extra auditing enabled beyond the default server logs).

Arc Agent on Linux Server

For the first test I used a Linux VM running in Google Cloud. Then I used the Azure portal to generate an install script for a single server.

To install the agent, I ran the script on the VM, and had to authenticate it by pasting a code generated in the terminal in a browser window. After this I was able to to connect to the VM from the Azure portal. Depending on what extension you install, there are multiple options for authenticating, including using Entra ID. I chose not to install anything extra, and added a public key to the VM, and then provided the corresponding private key to the Azure portal, to connect using plain SSH from an Azure Cloud Shell.

Detecting this connection is not straightforward, as Azure Arc sets up a proxy. Running the last command will only show a login from ::1, so it is not so easy to see that this is coming from an external attacker. If you try to use netstat to see live connections, it is also not immediately clear:

tcp        0      0 10.132.0.2:22           35.235.240.112:45789    ESTABLISHED 766092/sshd: cyberh
tcp6       0      0 ::1:22                  ::1:36568               ESTABLISHED 766163/sshd: cyberh

The top connection here was from the Google cloud shell, and the bottom one was the Azure Arc logon.

Since Linux logs sudo calls in the /var/log/auth.log file, we can see the installation commands being run:

Oct 23 07:56:33 scadajumper sudo: cyberhakon : TTY=pts/0 ; PWD=/home/cyberhakon ; USER=root ; ENV=DEBIAN_FRONTEND=noninteractive ; COMMAND=/usr/bin/apt install -y azcmagent
Oct 23 07:56:33 scadajumper sudo: pam_unix(sudo:session): session opened for user root(uid=0) by cyberhakon(uid=1000)
Oct 23 07:57:02 scadajumper sudo: cyberhakon : TTY=pts/0 ; PWD=/home/cyberhakon ; USER=root ; COMMAND=/usr/bin/azcmagent connect --resource-group quicktalks --tenant-id adf10e2b-b6e9-41d6-be2f-c12bb566019c --location norwayeast --subscription-id 8a8c2495-4b8c-4282-a028-55a16ef96caa --cloud AzureCloud --correlation-id fc67cc2f-7c66-4506-b2cf-2fe8cf618f25

If you suspect that an Arc agent has been installed, you can thus grep for “azmagent” in the auth.log file. From the last log line here we have quite a lot of data that helps us with attribution.

• Tenant-ID can help you identify the attacker’s Azure tenant. If you use the URL https://security.microsoft.com?tid=<tenant-id&gt; you may also get a logo if they have configured that.
• You also get the resource group and the Azure location used by the attacker

Another way to find this installation is to have a look at the running services.

systemctl list-units | grep -i azure
azuremonitor-agentlauncher.service loaded active running   Azure Monitor Agent Launcher daemon (on systemd)
azuremonitor-coreagent.service loaded active running   Azure Monitor Agent CoreAgent daemon (on systemd) loaded active running   Azure Monitor Agent daemon (on systemd)
himdsd.service loaded active running   Azure Connected Machine Agent Service

The himdsd.service is the service running the Arc agent. Overview of the Azure Connected Machine agent – Azure Arc | Microsoft Learn. In the documentation for the Azure Arc agent, we learn that there are log files generated in multiple non-standard locations. Performing an SSH-based login with private key from Azure Cloud Shell, we find the following lines in the log /var/opt/azmagent/log/hidms.log:

time="2023-10-23T11:56:35Z" level=debug msg="Connect Agent: read incoming message"
time="2023-10-23T11:56:35Z" level=debug msg="Connect Agent: pushing event to subscriber on microsoft.guestgateway"
time="2023-10-23T11:56:35Z" level=debug msg="Loading AgentConfig file from: /var/opt/azcmagent/agentconfig.json"
time="2023-10-23T11:56:35Z" level=info msg="config defaults set: &{ServiceConfigurations:[{ServiceName:SSH Port:22}] LocalAllowedPorts:[] ConnectionsEnabled:true EstsEndPoint:https://login.microsoftonline.com GnsURI:https://guestnotificationservice.azure.com ServiceBusEndpoint:.servicebus.windows.net SkipSSLCheck:false Location:norwayeast HeartBeat:60 EndPoint:/var/opt/azcmagent/socks/notify.sock LogLevel:INFO ResourceId:/subscriptions/8a8c2495-4b8c-4282-a028-55a16ef96caa/resourceGroups/quicktalks/providers/Microsoft.HybridCompute/machines/scadajumper}"
time="2023-10-23T11:56:35Z" level=debug msg="Loading AgentConfig file from: /var/opt/azcmagent/agentconfig.json"

time="2023-10-23T11:56:35Z" level=debug msg="HIS host name: gbl.his.arc.azure.com, IPv4 address: 20.50.1.196"

time="2023-10-23T11:56:35Z" level=info msg="cloud endpoint: &{Name:AzureCloud ImdsName:AzurePublicCloud ARM:https://management.azure.com ActiveDirectory:https://login.windows.net IdentityService:his.arc.azure.com GlobalIdentityService:https://gbl.his.arc.azure.com GuestConfigEndpoint: EstsEndPoint:https://login.microsoftonline.com PasEndPoint:https://pas.windows.net GnsURL:https://guestnotificationservice.azure.com ServiceBusEndpoint:.servicebus.windows.net GuestConfig:https://agentserviceapi.guestconfiguration.azure.com EndpointsToCheck:map[https://agentserviceapi.guestconfiguration.azure.com:true https://gbl.his.arc.azure.com:true https://login.microsoftonline.com:false https://login.windows.net:false https://management.azure.com:false https://pas.windows.net:false] Portal:https://portal.azure.com ArmMetadata:{Portal: Authentication:{LoginEndpoint: Audiences:[] Tenant: IdentityProvider:} Media: GraphAudience: Graph: Name: Suffixes:map[] DataplaneEndpoints:map[] Batch: ResourceManager: VMImageAliasDoc: ActiveDirectoryDataLake: SQLManagement: MicrosoftGraphResourceId: AppInsightsResourceId: AppInsightsTelemetryChannelResourceId: AttestationResourceId: SynapseAnalyticsResourceId: LogAnalyticsResourceId: OssrDbmsResourceId:}}"time="2023-10-23T11:56:35Z" level=info msg="Connect Agent: checking to see if connection is allowed with remote control plane data"

time="2023-10-23T11:56:35Z" level=debug msg="Loading AgentConfig file from: /var/opt/azcmagent/agentconfig.json"time="2023-10-23T11:56:35Z" level=info msg="config defaults set: &{ServiceConfigurations:[{ServiceName:SSH Port:22}] LocalAllowedPorts:[] ConnectionsEnabled:true EstsEndPoint:https://login.microsoftonline.com GnsURI:https://guestnotificationservice.azure.com ServiceBusEndpoint:.servicebus.windows.net SkipSSLCheck:false Location:norwayeast HeartBeat:60 EndPoint:/var/opt/azcmagent/socks/notify.sock LogLevel:INFO ResourceId:/subscriptions/8a8c2495-4b8c-4282-a028-55a16ef96caa/resourceGroups/quicktalks/providers/Microsoft.HybridCompute/machines/scadajumper}"
time="2023-10-23T11:56:35Z" level=debug msg="Loading AgentConfig file from: /var/opt/azcmagent/agentconfig.json"
time="2023-10-23T11:56:35Z" level=debug msg="HIS host name: gbl.his.arc.azure.com, IPv4 address: 20.50.1.196"
time="2023-10-23T11:56:35Z" level=info msg="cloud endpoint: &{Name:AzureCloud ImdsName:AzurePublicCloud ARM:https://management.azure.com ActiveDirectory:https://login.windows.net IdentityService:his.arc.azure.com GlobalIdentityService:https://gbl.his.arc.azure.com GuestConfigEndpoint: EstsEndPoint:https://login.microsoftonline.com PasEndPoint:https://pas.windows.net GnsURL:https://guestnotificationservice.azure.com ServiceBusEndpoint:.servicebus.windows.net GuestConfig:https://agentserviceapi.guestconfiguration.azure.com EndpointsToCheck:map[https://agentserviceapi.guestconfiguration.azure.com:true https://gbl.his.arc.azure.com:true https://login.microsoftonline.com:false https://login.windows.net:false https://management.azure.com:false https://pas.windows.net:false] Portal:https://portal.azure.com ArmMetadata:{Portal: Authentication:{LoginEndpoint: Audiences:[] Tenant: IdentityProvider:} Media: GraphAudience: Graph: Name: Suffixes:map[] DataplaneEndpoints:map[] Batch: ResourceManager: VMImageAliasDoc: ActiveDirectoryDataLake: SQLManagement: MicrosoftGraphResourceId: AppInsightsResourceId: AppInsightsTelemetryChannelResourceId: AttestationResourceId: SynapseAnalyticsResourceId: LogAnalyticsResourceId: OssrDbmsResourceId:}}"
time="2023-10-23T11:56:35Z" level=debug msg="Connect Agent: gwa: processing ingress request for SSH:22"
time="2023-10-23T11:56:35Z" level=debug msg="Connect Agent: Opened hc: a71fac9f-68a5-407b-be67-1370e650d34a, wsAddress: wss://g10-prod-am3-010-sb.servicebus.windows.net/$hc/microsoft.hybridcompute/machines/3ced11256d4ee0afd485e76e1b6ce6b287a684cb4d040e329c13529bc153d8c7/1698062195064991744/v2?sb-hc-action=accept&sb-hc-id=71bd5a85-f3d0-4636-8cbe-c7c73d2ec962_G10_G4, localConnection: {serviceName:SSH port:localhost:22 checkAllowedLocally:false}"
time="2023-10-23T11:56:35Z" level=debug msg="Connect Agent: Successfully added listener for conID: a71fac9f-68a5-407b-be67-1370e650d34a, target: localhost:22"

This log contains quite a lot of interesting inforamtion. If you want to search the log file for interesting events, grepping for “Connect Agent” is a good start.

Arc Agent on Windows server

When we try the same thing on Windows, it is a bit more obvious in sign-in events, because the authentication using the agent still generates Event ID 4624, and also shows SSH as the sign-in process.

In this case, we created a new username “hacker”, and used this to log in from Azure Arc in the portal, using hte Azure cloud shell. We see that the Logon Process is reported as sshd. On Windows this is easier to discover, as SSH is not commonly used for remote access. In fact, the SSH server is not installed by default, so we had to install it first, to make this technique work for remote access.

Also, when being logged in with GUI access, the Azure Arc agent will stand out both in the Start menu after being installed and in the list of installed applications. This test was run on a Windows Server 2022 running on an EC2 instance in AWS.

If you are not using Azure Arc yourself, and you see the “Azure Connected Machine Agent” among installed apps, there is reason to investigate further!

In summary

In summary, using Azure Arc as a persistence vector is interesting. On Linux it is more stealthy than on Windows, largely because of the localhost reference seen in the last log, whereas on Windows Event ID 4624 with sshd as logon process is typically reason to be skeptical.

The technique, when used as shown here, will leave traces of the Azure tenant ID and subscription ID, which can be used for attribution. On Linux it is interesting to note that the most useful log data will not be found in the /var/log folder, but in /var/opt/azmagent/log instead, a location where most defenders would quite likely not look for bad omens.