MOVEit vulnerability: Observability context fills log data gaps for MOVEit Transfer vulnerability

As organizations investigate and remediate the effects of the MOVEit vulnerability, they’re encountering gaps in log data and payload details. In many cases, this leaves teams unsure if they’re overlooking critical evidence of potential exploits of the MOVEit Transfer vulnerability. But observability context coupled with the right forensics tools, can fill those holes to detect otherwise hidden evidence of exploitation activity.

As teams conduct forensic investigations into the MOVEit Transfer vulnerability, many are finding they can’t conclusively determine their exploitation status. This uncertainty is because they may not have collected enough logs, or the logs don’t contain enough information. When working with logs originating from an out-of-the-box software such as Progress MOVEit Transfer, we can’t do much. What the software has been designed to log is what we get. But observability data (traces) can fill in the blanks to reveal useful evidence of possible exploitation, as proved by our analysis of the MoveIT vulnerability using Dynatrace.

What is the MOVEit vulnerability?

The MOVEit vulnerability is a critical SQL injection flaw in the web application MOVEit Transfer, which MITRE published as CVE-2023-34362 on May 31, 2023. When exploiting the vulnerability, attackers can gain remote code injection capabilities in the MOVEit server and modify or steal sensitive data from its database. On June 6, 2023, the ransomware gang Cl0p claimed responsibility for exploiting the MOVEit Transfer vulnerability starting on May 27, 2023, during the U.S. Memorial Day holiday. As a zero-day vulnerability, however, threat actors could have been exploiting the vulnerability since 2021, when the vulnerability was introduced into the software. Victims include large financial institutions, government agencies, and other critical service providers.

As soon as MITRE published the CVEs and the bad actors launched their exploits in the wild, the infosec community got busy reverse engineering malware and investigating attacker tactics. Progress Software, the company behind MOVEit softwareTransfer, has published an excellent overview, as have Huntress Labs, Mandiant, TrustedSec, Horizon3.ai and others.

Using Dynatrace to investigate the MOVEit Transfer vulnerability

Investigating the MOVEit vulnerability is a perfect use case for Dynatrace: having all your observability and security data immediately available to investigate a potential compromise, combined with indexless, schema-on-read storage and excellent processing speed, with one language (Dynatrace query language—DQL) to query them all.

To see how Dynatrace investigates the MOVEit vulnerability, we set up a proof of concept (PoC) to explore the exploit and its impact in a sandbox environment. The question was: Using Dynatrace Grail and DQL, what kind of attacker activity can we discover in the logs using an out-of-the-box logging setup both for the MoveIT vulnerability and the IIS logs of the server running the software?

As we started experimenting with the PoC and looking for the indicators of compromise (IoCs) in the IIS logs published by the community, the sky seemed clear. Running blazing-fast queries on historical logs stored in Grail was pretty straightforward. Lucky, because the vulnerability reportedly dates back to 2021. Because IIS logs contain IP addresses, client requests, and server response codes, querying them seemed quite promising to find suspicious indicators like certain IP addresses, strings and requests reported by the community.

For example, the LEMURLOOR web shell Cl0p used during the Memorial Day attacks used a backdoor called human2.aspx. This backdoor enabled attackers to access MOVEit data and users, download files, insert an administrative user, which enabled attackers to bypass credentials. The following query looks for the string human2.aspx in logs to detect if the attacker has added a webshell to exploit the MOVEit software.

The MOVEit vulnerability log quality wall

As the research progressed, however, we hit the log quality wall.

We found that the MOVEit default logs contained no useful information on the attacker’s activity.

The problem with the IIS server access logs was the headers. IIS access logs often log some headers but not all headers. So we’re likely to miss important evidence that would make the investigation and attack detection much faster.

And unfortunately, payloads containing valuable information on the exploits are not logged at all. Payloads would be nice, although the logs would get too crowded and full of privacy issues. But a security analyst can always dream…

These deficits left us unsure about what the attacker had done, their methods, and what data they compromised.

Using observability data for indicators of compromise

The disappointment with log quality sent us looking for more data to dig into. A good source for additional information is observability data. We can use metrics to detect anomalies, events to find unusual activities, and traces, which provide detailed insights into system activities. In Dynatrace, we can use one query language, DQL, to query all these different data sources.

For example, to see if the attackers installed a web shell, we can use DQL to find spans that contacted the endpoint human2.aspx.

In this query, we first fetch the spans (a span represents a single trace segment). Next, we filter them by the MOVEit service to improve the query performance. Finally, we filter by http.target that contains human2.aspx.

There are no results in this case, but this doesn’t mean there was no exploitation. There are other options to exploit the vulnerability. And since this IOC is now well-known, attackers will likely do it differently. You can normally find this information easily in logs (such as  IIS logs). But when you can get the information from spans, results get much more interesting.

Spans reveal more about MOVEit vulnerability exploit activity

Using spans, for example, we can find evidence of MOVEit vulnerability exploit activity in database queries. The exploit works by modifying the access tokens using SQL injection (SQLI). By using DQL to query spans, we can see if there were updates to the respective tables.

This is a simple query to check for updates on the hostpermits table, which is necessary to use the created access token from a remote host. The query is quite simple, but it might miss some matches. For example, if the database name is prepended or the table name has backticks around it. Let’s improve on the query to get better results and look for additional tables at the same time:

The improved query matches a wider variety of options and we can see that it found several SQL injections. The parse section of the query performs the following logic:

• DATA: matches any data, to match the beginning of the SQL query
• update: matches a string literal to find update statements
• BLANK: after the update keyword, there needs to be a space, but it could be more than one
• ('moveittransfer.'|'`'|''): matches either movittransfer (if the database name is prepended), a backtick (table names can be surrounded by backticks), or nothing
• ('hostpermits' | 'userexternaltokens'| 'trustedexternaltokenproviders'): matches against different table names

Logs, traces, spans – you need them all for investigations like the MOVEit vulnerability

It’s not possible to create logging from scratch for out-of-the-box software. We need to work with what we can get. Existing logs might not contain enough information to investigate the incident. Teams may have disabled useful logging by default, and when they discover the incident, it’s too late to configure logging. Therefore, looking at observability data jointly with logs can be a game changer.

Our investigation of the MOVEit Transit vulnerability proved that querying spans can give us valuable missing pieces to detect what the attackers were up to.

Interested in trying out these queries in your own environment? If you’re already a Dynatrace customer, these queries are available in a Dynatrace Notebooks template. Just ask your Dynatrace account rep.