Profile picture
F-Secure @FSecure
, 89 tweets, 42 min read Read on Twitter

THE HUNT: A Cyber Attack in the Process Industry

#cybersecurity #thehunt…
In manufacturing...

• 86% of cyber attacks are targeted

• 66% feature hacking

• Only 34% involve malware
In manufacturing, almost half – 47 percent – of breaches involve the theft of intellectual property to gain competitive advantage.

#cybersecurity #thehunt
In manufacturing, 53% of the attacks are carried out by state-affiliated actors and 35% by organized crime.…

#cybersecurity #thehunt
Meet our victim...

Pulp Global.

Pulp Global Inc. is a leading producer of various types of pulp products

Number of employees: 12,500

#cybersecurity #thehunt…
Pulp Global has been growing quickly over the past five years, acquiring many smaller competitors.

Martin, the CIO of Pulp Global, has a plan.

He wants to roll out reporting and manufacturing execution systems to all 25 of the company’s mills.

That’s where our story begins.
Pulp Global is under extraordinary pressure:

•Their industry is a highly competitive
•Producers need the lowest possible costs
•There are as many different ways of building and running a mill as there are mills
•Only 5 of its 25 mills use the same systems

And don’t forget about growth.

NEWS: Pulp Global to build $500 million pulp mill in the Samveng delta of Southeast Asia.

Investors like it. But guess who also reads the news?
Number One is the leader of an organized cyber crime group.

Number One and his team perform sophisticated, targeted attacks against companies around the world.

They extort money, sell confidential information on the black market or create reputational and financial damage.

Number One decides to do a little investigation of Pulp Global.

He calls this his “get-familiar phase.”

His team looks into vendors, suppliers, maintenance contractors, personnel, and information about the IT and OT systems.

One word keeps popping up: Segnen.
The Segnen mill is Pulp Global’s largest.

Built on industrial estate along the coast of the Baltic Sea, it has:

* 5 distributed control systems
* 8 SCADA systems
* 3 mill information systems
* And that’s just the beginning of the complexities…
And there it is—a treasure trove in the shape of a 130Mb zip folder.

The files inside:
•Mill floor plans
•A list of equipment with brand, models & version #s
•A list of new software, OSes, network layouts, IP addresses, Modbus tags and I/O listings, and even and passwords
How did they find this treasure trove?

An engineer from Pulp Global’s third party integrator had received an email from containing only the zip file with the subject: “Important project documents.”

He decided to upload the zip file into a reputable malware scanning service.
How did an organized crime group get a zip file from a reputable malware scanning service?

Malware researchers occasionally have ties with the underground crime community.

#cybersecurity #thehunt…
Now, they just needed an employee to target.

Meet Peter.

Peter has just finalized his Master’s degree in Automation Technology and was almost immediately hired by Pulp Global.

His thesis just happens to be about – wait for it! – the Pulp Global's plant automation project.
With the plant and an employee to target chosen, Number One's team leaps into action.

The bad news for these hackers?

The Segnen plant hasn’t been modernized and has a limited attack surface.

Very limited.

#cybersecurity #thehunt…
But there was an email internal newsletter service in the cloud.

Smart move for lowering costs, but there’s a problem.

The service also had a test environment where the service developers could test & deploy changes.

And there is an admin account

Login: admin
Password: admin
With a few keystrokes, Number One used the admin/admin login to siphon all the email addresses, names and titles of Pulp Global’s employees, as well as the templates used to send internal newsletters.


#cybersecurity #thehunt
FYI: Here’s why you need to protect your credentials.
#cybersecurity #thehunt

Number One decides it is time to launch a targeted spearphishing campaign.

Through the newsletter, he learns employees recently had been rewarded with a party.

He creates an email using Pulp Global’s template...

Subject: Employee Celebration Pics
The spearphishing email contained a zip folder with a custom-built remote access trojan (RAT).

Once executed, it would connect back to Number One’s command and control infrastructure and allow him to perform additional attacks.

#cybersecurity #thehunt…
The targets for the spearphishing campaign are strategically selected to include Peter, the new hire who did his thesis about automating Segnen.

Peter couldn’t go to the party but he’s interested in seeing what he missed.

Official template. Seems legit.

He clicks.

The attached file promptly infects Peter’s laptop with the RAT, remote access trojan.

It only takes about an hour from sending the email to getting the initial foothold via Peter’s compromised computer.

Game ON.

#cybersecurity #thehunt…

Number One and the team spend the next two weeks mostly celebrating. No further attacks or trying to extend their foothold.

Instead, they monitor Peter’s laptop, logging keystrokes and capturing screenshots of the systems Peter accesses.
Luckily for Number One and his team of hackers, Peter leaves his laptop on – all the time.

This allows the criminals to utilize Peter’s PC outside office hours.

They probe the network, launch attacks and extend their foothold.

#cybersecurity #thehunt…
One of the first things Number One probes is the project management system and wiki.

The amount of information and the level of detail he finds are unlike anything he’s come across before.


#cybersecurity #thehunt
Over the next few days, the hackers download every...

*network drawing
*system & shop floor layout
*project plan
*equipment list
*asset inventory report.

Their exfiltration of information detailing Pulp Global’s projects and sites totals tens of gigabytes.
Now Number One’s team splits up the work.

Some comb through the millions of pages of documents. Others try to extend their foothold in Pulp Global’s network.

To remain undetected, they use all the same tools the company’s IT staff and system administrators use on a daily basis.
But the hackers are working too hard.

Pulp Global has no network-level visibility and very limited host-level visibility.

The attackers can run their tools, no matter how noisy, without a significant risk of getting caught.

#cybersecurity #thehunt…
Here’s what's wrong with Pulp Global's networks:

1.They're a mishmash built over 3 decades
2.They include everything from legacy applications and servers to new Windows systems
3.Many systems receive no updates, making them easy to infect with publicly available exploits
Uh oh. There it is.

In a photo from IT support staff member, a computer monitor in the corner has a
Post-It note stuck to it.

The note contains the domain administrator credentials used by Pulp Global’s IT support staff.

Number One doesn’t believe they will work. But they do.
The hackers have now owned the entire network.

These credentials were reused so much within the corporate network that they also worked for Linux hosts, and even the routers and switches.

#cybersecurity #thehunt…

Meet Eric, a control systems engineer working for a third-party integrator.

He’s guy who uploaded the zip file to the reputable malware scanner

Remember that?

Why is Eric is a target?

He’s responsible for the newly installed manufacturing execution system (MES) used to synch Pulp Global’s enterprise resource planning
(ERP) system to production.

MES also automates the flow of data between ERP and production.

#cybersecurity #thehunt
From the network drawings and mappings of the network equipment, Number One establishes that the MES system is the key link.

It has perfect access to both the corporate and production Incident Command System networks.

#cybersecurity #thehunt
With nearly unlimited access to the various production networks and systems, Number One’s group focus on the Segnen site.

#cybersecurity #thehunt…
Number One’s group probes the Segnen plant to map…

*the different systems
*their software versions
*the PLCs (programmable logic computers)
*the DCS (distributed control system) equipment used in the mill

And for fun, they try to harvest more user credentials

Further recon establishes that production Active Directory is using SMBv1 – a legacy protocol that makes it possible for the attackers to query for user accounts, account descriptions and hosts.

BONUS: Local admins had written in the passwords for easy access for the employees.
Most of the networks in the mill are on the same network.

However, one network is separate -- the engineering workstations that program the programmable logic computers (PLCs) and safety instrumentation systems (SIS) used to run the plant.

It takes a huge effort from Number One’s group to locate the engineering workstations among the thousands of hosts.

Eventually, the engineering workstations is found via an unlikely source -- an operator station sitting in one of the control rooms.

#cybersecurity #thehunt
The operator station has recently been upgraded with a second network interface.

This makes it possible for automation engineers to access the engineering workstations while monitoring operations from the control room.

It's also the hackers' way in.…
This “dualhome” control room host is Number One’s gateway to the engineering workstations. But it doesn’t connect to the internet.

So the code and files are piped via multiple hops.

The hops end in the corporate network and on to the command-and-control server… of Number One.

After about a month of collecting information and extending the foothold in Pulp Global’s networks, Number One finally has enough information to prepare the targeted attack on the industrial control systems at the Segnen mill.
But now they have to learn pulping.


In addition to generic resources, such as Google and various pulp industry trade publications, the attackers pour over Peter’s thesis on plant automation to plan the attack.

#cybersecurity #thehunt…
The attackers learn that due to the complexity and fluctuations of the pulping process, any changes could take up to a week to show up in the finished product.

This means they can test and validate their attack on the production systems with only a minor risk of getting caught.
Despite the hackers' painstaking research, the group still lacks crucial details necessary to carry out a successful attack.

It’s clear they need additional help – on the inside.

#cybersecurity #thehunt…
Number One begins approaching Segnen mill employees by email and phone.

Playing the part of a new employee at one of the mill’s contractors, Number One asks questions that could only come from someone who knows the mill intimately.

The employees suspect nothing.

BTW: This is what's known as Social Engineering.
Armed with valuable information obtained from employees, Number One decides on the attack vector.

If successful, his attack would cause lasting reputational damage and financial loss.

If it fails, it will wipe out all PLCs, servers and hosts in the production environment.
To test the attack plan, Number One builds a simulated test environment.

Using the original source code, project files and HMI graphics, the group sets up a crude test bench for simulating and testing their attack payloads.

#cybersecurity #thehunt…
In order to distribute the attack payload, the code needs to be downloaded onto the PLCs & safety controllers.

This is easier said than done.

Segnen mill runs 24/7. Controllers can’t be taken offline or stopped for the code change without the attackers calling themselves out.
The hackers come up with a… hack.

By changing small parts of the specific code blocks and extending the existing variable in data blocks, the controllers don’t need to be stopped.

The new code can be downloaded onto the controller and will start running during the next cycle.
Number One and his hackers write and test multiple configurations and ways of conducting the attack.

They change six different PLCs, a few safety logics, and create new views for HMI panels used by the operators to monitor the process.

It’s go time.…

To maximize the impact, and make it extremely hard for the Segnen mill workers to pinpoint the problem, Number One attacks multiple parts of the pulping process using normal fluctuations to cloak the hack.

#cybersecurity #thehunt…
Here's how the hack works:

Temperatures used in the pulp cooking process begin to vary random intervals.

The fluctuations in temperature distort quality control system readings, which results in waste, quality problems, fines and -- ultimately --reputational damage.

Three months after the initial breach of Pulp Global’s systems, Number One finally launches the carefully crafted attack.

#thehunt is on.

#cybersecurity #thehunt…
But there's a problem.

Due to recent attacks against the safety controllers, Eric, the control systems engineer, implemented a physical key attached to the controllers, putting them on run-only mode.

This stops anyone making software changes without actually turning the key.
Number One and the crime group go back to the drawing board to figure out a way to fool the safety controllers.

The attack code is left running.

#cybersecurity #thehunt…
Meanwhile, the Segnen mill begins to experience strange problems with the quality of the pulp and arbitrary fluctuations in the control system parameters.

The control systems engineers, including Eric, work overtime to find the cause of the random fluctuations.

Then a vendor calls.

A quality control system is running a hidden process.

That shouldn’t be happening.

The vendor rep sends the executable file to the Pulp Global IT team and a security company it trusts.

#cybersecurity #thehunt…
The security company finds a purpose-built attack tool.

The pros recommend a thorough sweep of the production hosts to see if any other implants are hidden in the network.

Eric contacts his boss, who notifies the CIO of Pulp Global:

The mill is under attack.

Thinking this must be another case of commodity malware from an employee’s laptop or a contractor’s removable media, the CIO tells Eric, "Just keep on like normal."

#cybersecurity #thehunt…
Eric won’t give up. He checks out other hosts.

There it is: a huge number of alerts caused by one of the safety controllers on a valve.

But he can’t figure out why. Leaving the development environment open, he exits the control systems lab, puzzled.

#cybersecurity #thehunt
A few days later, Eric still can’t shake the feeling that something is wrong.

Normally reliable systems are now acting up, seemingly without any reason.

#cybersecurity #thehunt…
Meanwhile, Number One has also logged on to the engineering workstation used to program the safety controllers.

He saw the development environment open with the very same part of the code they’d been trying to circumvent for the better part of a week.

“How did they find us?”
Now the hackers have to act quick.

What could the group still do to inflict maximum damage with the foothold they still had?

“Time to say bye-bye to the Segnen mill operations,” Number One says, as he sends a command to all of the compromised hosts.

#cybersecurity #thehunt
The hosts begin running commands to wipe as many other hosts as possible.

Within two hours, the Segnen mill is completely halted.

The pulp cooking process has to be taken down using a manual override -- a process that can take weeks to reverse.…
But this is the least of Pulp Global’s worries.

Almost all of the Windows hosts in the mill are wiped clean.

Nearly 50 controllers, from two of their main vendors, suffer a similar fate.
Only a few controllers are unaffected.

#cybersecurity #thehunt…
Somehow the situation gets even worse.

Bored mill personnel begin posting pictures of blank control room displays on Facebook.

Reporters notice.

Eventually the news of the complete breakdown at the Segnen mill reaches the CIO–from a reporter looking for a comment.


After the call from the reporter, the CIO contacts Segnen to find out what's up.

The mill is in such chaos that it it’s impossible to get anyone to stay on the phone.

Pulp Global had no plans or procedures to guide the CIO in dealing with the incident.
The CIO decides to call in external help to handle the situation.

He asks the vendors supplying the Segnen mill to send first responders on-site immediately.

#thehunt for the hackers is on.

Just when the CIO thinks things couldn’t get any worse, Pulp Global’s CEO calls.

He just got an email.

“Segnen mill was just the beginning,” it reads. "Comply with our requests by the end of the day or this could get very expensive for you.”

#thehunt #cybersecurity
Pulp Global alerts the authorities and the local CERT. A cyber security company is called in to help contain the attack.

Meanwhile, the board of directors have called an emergency meeting about the ransom demand.

#thehunt #cybersecurity…
Forensic analysis is difficult. Log files and other useful information are almost impossible to find.

The analysts soon abandon the Segnen mill and turn their focus on the corporate network, as law enforcement investigates to see if it's an inside job.

#thehunt #cybersecurity
With just four hours to go until the ransom deadline, the forensics team discovers that the corporate Active Directory was completely taken over by the attackers.

This means the hackers have nearly limitless possibilities to attack other targets within Pulp Global.

The forensics team, with the help of Pulp Global’s IT, begin to isolate and segment the network.

They aim to limit the access of the adversaries while being careful not to draw attention to their efforts.

#thehunt #cybersecurity…
By the time the ransom deadline expires, the responders are confident that they had managed to contain the attack enough to avoid paying the ransom.

An hour after the ransom deadline passes, nothing has happened.

#thehunt #cybersecurity
Two hours after the deadline, the forensics team notices someone accessing the MES system server using domain administrator credentials through a VPN.

A quick check confirms it isn’t anyone within Pulp Global.

The adversaries have returned.

The forensics analysts log the adversaries’ actions and every network packet the hackers send.

The adversaries use a process running with system privileges to send commands to hosts within another mill’s production network.…

#thehunt #cybersecurity
Investigators find the same remote access trojan deployed to several other machines. Now the responders know what to look for.

Even better, the trojan keeps an easily decryptable log file of all of actions performed.

#thehunt #cybersecurity…
As the incident response and forensics investigations continue, the CEO and board try to come up with a communications plan.

But the problems keep growing…

Pulp Global’s share price has plummeted more than 10% within a day.

#thehunt #cybersecurity…
Uh oh.

Unhappy customers begin to demand compensation from Pulp Global due to the delays.

The local authorities investigate the environmental, safety and health hazards caused by the abrupt stop of mill operations.

#thehunt #cybersecurity
After the attack is contained, it takes Pulp Global nearly three months to recover all operations and eradicate all backdoors from multiple hosts.

It takes nearly a year to regain the confidence of customers and investors and deal the resulting the safety and health hazards.
When everything is added up, the attack ends up costing Pulp Global a large percentage of their revenue.

Although the direct cost of the forensics, equipment and labor are in the millions, the biggest cost can’t easily be measured -- the loss of customer & investor confidence.
To review #thehunt:

With cyber attacks growing more advanced by the day, you cannot rely on stopping all threats before they infiltrate your infrastructure.

These days, a compromised network is not a question of “if,” but “when” and “what do we do about.
The earlier in the cyber kill chain you detect an intruder, the less damage .

Speed matters.

A high-quality detection and response solution slashes attack detection from months to minutes.

Find out more about F-Secure’s Rapid Detection & Response:…
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to F-Secure
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member and get exclusive features!

Premium member ($30.00/year)

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!