Thread by @tralfamadorenik

THREAD: Were the DNC emails taken from a backup server?

The Mueller Netyksho indictment asserts that the DNC emails, subsequently dumped by WikiLeaks (first on July 22, 2016) were acquired by Russian GRU agents who leveraged a hack of the DNC.

The indictment is vague on details raising many questions as to how exactly the emails were acquired, ex-filtrated and then transmitted to WikiLeaks.

As we’ll see below, even the on-scene experts who investigated the incident seemed to come up empty handed.

Before we cover the details, let’s have a look at the timeline. This is from the Forensicator’s analysis of the DNC emails.

There are 2 acquisition dates on the timeline: May 22-23, 2016, and May 25, 2016. The indictment mentions only the May 25 acquisition date. Between those two dates, on the evening of May 23 and the morning of May 24 the DNC’s computer systems were shut down with minimal notice.

Prior to the time frame shown on that timeline, on May 5 the DNC brought in an outside consultant, Crowdstrike to investigate suspicious activity detected in the preceding month (April).

The president of CrowdStrike, Shawn Henry, said they had no concrete evidence of how the emails were acquired. In this excerpt, Henry mentions an event that preceded the email acquisitions.

The date, April 22, precedes the date of the email acquisitions by a month and Henry refers to documents (not emails) that were suspected to have been taken by a hacking group named Fancy Bear believed to be associated with the Russian GRU (military intelligence) agency.

Henry mentions that they saw evidence that some documents were “staged”, but they had no evidence that they were taken. A way to think about this – CrowdStrike found a zip archive of documents on a DNC server but was unable to determine if the documents had been taken.

When the questions turn to the topic of the DNC emails, Henry responds: “there is not [sic] evidence that they [the DNC emails] were actually exfiltrated.”

The highlighted statement where Henry says that the hackers might have acquired the emails by monitoring the traffic back/forth with the DNC Exchange email server is a false trail.

For one thing, the earliest email is dated January 2015, a year and a half before the incident; the monitoring would have to have been in place for over a year and it wouldn’t have been able catch the Sent folder nor personal file folders.

Further, the timestamps of the email files show acquisitions on May 22-23 and May 25, 2016.

Mr. Henry is just blowing smoke.

However, on the topic of email monitoring, we’ll show below, Mr. Henry might have been correct, just for a different reason than he intended.

NOTE: Henry does not mention any evidence that the DNC *emails* were “staged”. Per Henry, Crowdstrike finds no trace of evidence as to how/when the DNC emails were acquired.

ANALYSIS: How can it be there was no evidence of the alleged hack? How did 40,000 DNC emails end up on WikiLeaks, yet no one seems to know by what means they got there?
Here, we’ll consider an alternative theory that might explain how DNC emails were acquired without detection.

The short answer: CrowdStrike was looking in the wrong place. They should have looked at a server located in the Pacific timezone attached to the DHC virtual private network. Further, the perpetrators may have accessed a backup email server and not the main DNC server.

Per this theory, the logs which recorded access to the DNC Exchange server might have been found on a server that was physically distant from the DNC’s headquarters.

To support our theory, we’ll show (1) indications of a DNC email server running with Pacific Time Zone settings and (2) evidence that the DNC operated a backup server that served as a real-time mirror of the DNC Exchange server.

Forensicator showed that the May 25 batch of emails may have been exported from a system that had its clock set to the Pacific Timezone. See the excerpt below for details.

That’s not the only Pacific timezone indication. Additional indications can be found in the DNC email headers.

The DNC email collection on Wikileaks was a group of approximately 40,000 individual files. Each file records a single email message, recorded in EML format, described here: .loc.gov/preservation/d…

Fairly recently both the DNC emails and Podesta emails found on WikiLeaks were taken offline. WikiLeaks has been saying for a while now that the email collections will be restored. So far, they remain missing. This analysis refers to an archival copy of the emails.

For this analysis, we’ll group the DNC emails into the following categories: (1) an email filed into the Sent folder of the sender; there are no Received headers in those emails, (2) an external email originating from a non-DNC sender;

those emails are front-ended by a service known as AppRiver, (3) a normal email originating at the DNC, (4) an email originating at the DNC sent from an email server that seems to handle local mailing lists; that server is named “postman” and appears to be running Unix.

This chart shows this categorization of DNC emails.

First, let’s focus on the regular (local) emails that originate at the DNC. About 70% of the emails fall into this group. When we review one of the local emails we’ll notice something unusual about the “Date” field.

Usually, the Date: field will reflect the timezone of the Sender and the time shown will be the sender’s local time.

We expect emails that originate from the DNC headquarters (in Washington, DC) to reflect an Eastern timezone in the sent “Date” field, yet all of them show a Pacific timezone setting. Why?

We conclude that local emails are first routed through a server that has its clock set to GMT-7. That timezone setting applies to Pacific Daylight Savings Time and most of Arizona during April/May of 2016.

Let’s turn to an incoming (external) email from a non-DNC source.

This email exhibit was edited to show only the relevant headers, various identifying data has been anonymized.

AppRiver accepted all incoming DNC email and forwarded it to the DNC. AppRiver provided spam filtering and provided high availability. The sender’s date field shows a timezone offset of GMT-5, confirmed by the area code shown (Dallas, TX).

For DNC internal emails we expect to see a GMT-4 (EDT) timezone offset, yet we universally see GMT-7. This implies that all internal emails were routed through a server that had its timezone set to GMT-7 during the late spring of 2016.

If we tie this indication of a server with a GMT-7 timezone offset with the GMT-7 indications found in the last modification times of the May 25, 2016 email acquisition, we wonder: Were the emails exported from an internal DNC email server possibly located in the Pacific TZ /AZ?

About 3% of the DNC emails traverse a server known as postman[.]dnc[.]org. Those emails had a date offset of GMT+0000, which is typical of many email servers.

This confirms that when an email traverses a server, its date/time might reflect the TZ setting of the server.

NEW: Based on a new discovery, we have learned that the DNC ran a mirrored, real-time backup email server, described here.

Let’s return to the DNC internal email and add additional mark ups.

Item #1 is the new observation. We see that the email traversed a server named “DNCDAG1”. What does the phrase “DAG” suggest? In Exchange parlance: a DAG is a “Database Availability Group”.

A DAG server mirrors the Exchange database in real-time. If the main server goes down, Exchange can switch to a DAG backup server without a hiccup.

Did Mr. Henry make a Freudian slip when he suggested that the DNC email server could be monitored?

The following diagram shows how DAG servers fit into the Exchange architecture.

NOTE: We searched the DNC email messages for headers that referenced either DNCDAG1 or DNCDAG2. DNCDAG2 appears only during the last two days the leaked emails timeline.

We’ve shown the existence of an operational real-time DNC Exchange server mirror. This is sufficient to suggest that the WikiLeaks DNC emails might have been exported from a redundant server – in this case, DNCDAG2 could be that alternative server.

We think it was likely that DNCDIAG2 was operational throughout 2016 (and before). Thus, the emails could have been taken from the backup server at any time without touching the main DNC server.

If DNCDAG2 ran on a physically different server in a different location, perhaps on the West Coast or AZ then this easily explains how Crowdstrike might have missed this intrusion.

The DNC DAG timeline suggests that the main DNC Exchange server (DNCDAG1) was shut down or disconnected soon after the emergency shutdown. Why? As expected, we see that DNCDAG2 took over.

NOTE: Various reports indicate that Crowdstrike made image copies of various DNC servers; some/all of that forensic data was made available to the FBI. The testimony we reviewed doesn’t indicate when those server images were collected, or which server images were collected.

Did Crowdstrike image only the servers found at DNC Headquarters in Washington, DC? If the DAG server(s) were missed or a possible West Coast server was missed might that explain why both Crowdstrike and the FBI didn’t find signs of when and how the DNC emails were acquired?

We wonder if the DNC first noticed the May 22-23, 2016 access of the DNC Exchange server and then decided to perform an emergency shutdown to clear intruders and diagnose the system compromise?

Perhaps out of caution the DNC system administrators shut down the main server and let DNCDAG2 take over? Was the DNC staff (and presumably Crowdstrike) surprised again when the May 25, 2016 exfiltration occurred?

Or did they read about that in the papers when WikiLeaks disclosed the DNC emails on July 22, 2016?

END

Netting things out, it seems likely that the backup server had its clock set to GMT-7, placing it in the Pacific Daylight Savings or Arizona (MST) time zone.

The next question is: how were they able to gain access to this server?

ADD: In this post, it was suggested that the emails traversed a system that had a GMT-7 time zone offset.

Instead, it seems more likely that the GMT-7 setting reflects the TZ of the system from which the emails were exported as EML files.

https://twitter.com/tralfamadorenik/status/1766514881413443921

@threadreaderapp please unroll.