Part 6 — What I Changed After the Restore

• Leave a comment

The final practical part of a real-world WordPress VPS compromise investigation

By SepedaTua — CrushEdge.com

The restore was the easy part.

The harder question was:

How do I make sure I don’t have the same problem again next week?

I could restore the websites from a known-good backup, start Apache and MariaDB, and call it a day.

That would be a mistake.

A backup gets you out of trouble.

It doesn’t fix the door that allowed somebody in.

So after the restore, I approached the server in layers.

1. Don’t immediately put everything back online

This is probably the least exciting advice in this entire series.

It’s also one of the most useful.

If you’ve restored a compromised VPS, don’t do this:

systemctl start mariadb
systemctl start httpd

and immediately go to bed.

Instead, verify the environment first.

I want to know:

Is the filesystem clean?
Are the expected websites present?
Are the databases present?
Are unexpected users present?
Are unexpected services listening?
Is MySQL exposed?
Are there suspicious cron jobs?
Are SSH keys correct?

Only then do I bring services back.

2. Check the server’s listening ports

This is one of my first commands:

ss -lntup

I want to know what the server is actually listening on.

Typical output might include:

22
80
443
10000

depending on how the server is configured.

The important question is not:

“Are these ports dangerous?”

The important question is:

“Do I know why every one of these ports is open?”

If you find something you don’t recognize, investigate before exposing the server again.

3. MySQL was the part I wanted to fix properly

This deserves special attention because I had legitimate applications that needed remote database access.

The lazy solution is:

MySQL
   ↓
3306
   ↓
Internet

and then create:

'user'@'%'

It works.

It is also much broader access than most applications actually need.

4. Why % is a problem

In MySQL:

'user'@'%'

basically means:

this MySQL account isn’t restricted to a particular source host by the MySQL account definition.

That does not automatically mean anyone on the Internet can log in.

You still need:

  • MySQL listening on a reachable interface
  • firewall access
  • valid credentials
  • compatible authentication

But combining:

3306 exposed
+
'user'@'%'
+
password

is a configuration I don’t want unless there is a very good reason.

5. The problem with a temporary VPS

This is where things get interesting.

Sometimes I need another VPS temporarily.

For example:

Main server
     ↑
     │
Temporary cloud VPS
     │
     └── runs a script for a few hours

Then I destroy the temporary VPS.

Its public IP changes every time.

So whitelisting:

203.0.113.45

is inconvenient.

But that doesn’t mean I need to expose MySQL to the entire Internet.

There are better options.

6. My preferred solution: SSH tunneling

If the temporary server can SSH into the main server, I don’t need to expose MySQL publicly at all.

Keep MySQL listening locally:

127.0.0.1:3306

Then create an SSH tunnel from the temporary VPS.

Conceptually:

Temporary VPS
     │
     │ encrypted SSH
     ▼
Main VPS
     │
     ▼
127.0.0.1:3306
     │
     ▼
MariaDB

The temporary VPS can then connect to its own local forwarded port.

For example:

ssh -N \
  -L 13306:127.0.0.1:3306 \
  mainuser@main-server

Then the application on the temporary VPS connects to:

127.0.0.1:13306

instead of:

main-server:3306

That’s a much nicer arrangement.

7. And the temporary VPS can still have a changing IP

That’s the nice part.

The tunnel doesn’t require me to maintain a permanent firewall whitelist for the temporary VPS.

The connection is:

temporary VPS
        ↓
SSH authentication
        ↓
main server
        ↓
local MariaDB

When I’m finished:

exit

or terminate the SSH process.

Then destroy the temporary VPS.

No permanent MySQL Internet exposure required.

8. If I absolutely need direct MySQL access

Sometimes an SSH tunnel isn’t practical.

Then I would use a dedicated MySQL account.

Not the root account.

Not the application owner’s administrative account.

Something like:

CREATE USER 'temporary_reader'@'%' IDENTIFIED BY 'A-LONG-RANDOM-PASSWORD';

Then grant only what the temporary application needs.

For example:

GRANT SELECT
ON my_database.*
TO 'temporary_reader'@'%';

If it only needs to read data, don’t give it:

INSERT
UPDATE
DELETE
DROP
ALTER
CREATE
GRANT OPTION

The account should have exactly what the application requires.

9. Then destroy the temporary account

This is the part people forget.

When the temporary VPS is gone:

DROP USER 'temporary_reader'@'%';

Don’t leave:

temporary_reader

sitting around forever because:

“Maybe I’ll need it again.”

I know that habit.

I’ve done it myself.

It’s how temporary things become permanent security problems.

10. If you want direct access, use a VPN

For repeated access between machines, I’d prefer a private network.

For example:

Main VPS
   │
   │ private VPN
   │
Temporary VPS

Then MariaDB can be bound to the VPN interface/address rather than the public Internet.

This is especially useful if multiple servers need database access.

For a VPS that exists for only a few hours, though, SSH tunneling is usually simpler.

11. Check the MySQL accounts after restoration

After starting MariaDB:

mysql -uroot -p

then:

SELECT User, Host
FROM mysql.user
ORDER BY User, Host;

Look specifically for:

%

and unexpected users.

For example:

wordpress_user    localhost
wordpress_user    %

The second line deserves attention.

Maybe it is intentional.

Maybe it isn’t.

12. Find accounts with broad privileges

You can inspect grants:

SHOW GRANTS FOR 'wordpress_user'@'%';

For every remote account you don’t recognize, inspect it.

You can also look for powerful privileges:

SELECT User, Host, Super_priv, Grant_priv, Create_user_priv
FROM mysql.user;

I don’t want an ordinary application account with unnecessary administrative privileges.

13. Don’t use MySQL root from applications

This should be obvious, but I’ve seen it many times.

Don’t put:

mysqli_connect(
    "localhost",
    "root",
    "password",
    "database"
);

inside a website.

Create a database-specific account instead.

For example:

CREATE USER 'shop_app'@'localhost'
IDENTIFIED BY 'LONG-RANDOM-PASSWORD';

GRANT SELECT, INSERT, UPDATE, DELETE
ON shop_database.*
TO 'shop_app'@'localhost';

If the application doesn’t need schema changes, don’t give it:

CREATE
ALTER
DROP

14. WordPress uploads deserve special attention

One of the things that stood out during this incident was the use of files pretending to be images.

A good defensive layer is preventing PHP execution from the uploads directory.

If you’re running Apache, you can use an appropriate configuration in:

wp-content/uploads/

to prevent PHP scripts from executing there.

The exact configuration depends on whether PHP is running through Apache modules, PHP-FPM, and your Virtualmin configuration.

Don’t blindly paste a random .htaccess rule from the Internet.

First determine how your PHP handler is configured.

The principle is:

User uploads image
        ↓
stored as file
        ↓
cannot execute as PHP

That’s much better than trusting the extension.

15. Don’t make uploads writable more broadly than necessary

WordPress needs to write to:

wp-content/uploads

It doesn’t mean the entire WordPress tree needs to be writable by the web process.

A good rule is:

Only make the directories that actually need writing writable.

This makes an attacker’s job harder.

If PHP can modify:

wp-admin/
wp-includes/
wp-content/plugins/

during normal operation, an attacker who gains PHP execution has more opportunities to modify core code.

16. WordPress core should be boring

This is what I want:

wp-admin/
wp-includes/
wp-login.php
wp-settings.php
...

to look like normal WordPress.

Not:

wp-admin/
    random123.php
    helper-old.php
    cache2.php

and certainly not:

wp-includes/images/
    something.png

where file says:

PHP script

A clean WordPress installation should have a predictable structure.

That makes anomalies easier to spot.

17. Keep WordPress updated

After restoration, check:

wp core version \
--path=/home/customer01/public_html

Then:

wp plugin list \
--path=/home/customer01/public_html

and:

wp theme list \
--path=/home/customer01/public_html

Don’t blindly update everything on production without testing.

But don’t run an ancient plugin forever because:

“It still works.”

That’s how vulnerabilities become permanent infrastructure.

18. Remove things you don’t use

This includes:

  • unused plugins
  • unused themes
  • old test scripts
  • development copies
  • abandoned WordPress installations
  • forgotten staging sites

Every installed application is another piece of software that can eventually become vulnerable.

If I don’t use it:

delete it

is usually better than:

leave it disabled forever

19. Staging sites need the same security attention

One thing I learned from this incident is that people often protect the main site and forget:

staging.example.com
test.example.com
dev.example.com
old.example.com

Attackers don’t care which site you consider “important.”

If a vulnerable staging site lives on the same VPS, it can become the entry point to the server.

So:

Treat every web-accessible site on the VPS as potentially dangerous.

20. Check for forgotten PHP files outside WordPress

After restoration, I like:

find /home \
-type f \
\( -name '*.php' -o -name '*.phtml' -o -name '*.php5' -o -name '*.php7' \) \
-printf '%u:%g %m %s %p\n' \
2>/dev/null |
sort

This isn’t about deleting everything unfamiliar.

It’s inventory.

The question is:

Do I know what all these applications are?

21. SSH deserves its own cleanup

Check:

sshd -T | grep -Ei \
'permitrootlogin|passwordauthentication|pubkeyauthentication'

If possible, I prefer:

public-key authentication

over password authentication.

And if I don’t need direct root SSH login, disable it.

But do this carefully.

Before changing SSH configuration:

keep the current SSH session open

Open another session and test the new configuration.

Don’t lock yourself out of your own VPS at 2 AM.

Ask me how I know.

22. Change credentials after a compromise

This includes:

VPS / system

  • root password
  • sudo users
  • SSH keys

Virtualmin

  • administrator passwords
  • API credentials

WordPress

  • administrator passwords
  • application passwords
  • API credentials

Database

  • database users
  • remote database accounts

External services

  • SMTP
  • Cloudflare/API tokens
  • backup credentials
  • Git credentials
  • deployment keys

If a secret was stored on the compromised server, assume it might have been readable.

That’s the safe assumption.

23. Don’t forget API keys

This one is easy to overlook.

Search your application configuration for things like:

API_KEY
SECRET
TOKEN
PASSWORD
AWS_
CLOUDFLARE
SMTP

For example:

grep -RInE \
'API_KEY|APIKEY|SECRET|TOKEN|PASSWORD|AWS_|CLOUDFLARE' \
/home/customer01/public_html \
2>/dev/null

This will produce false positives.

That’s fine.

We’re looking for secrets that need rotation.

24. Backups need to be protected too

A backup sitting on the same compromised VPS isn’t a great backup.

Ideally:

Production VPS
       │
       ▼
Backup storage
       │
       ├── separate machine
       ├── restricted access
       └── historical copies

And importantly:

Keep multiple restore points.

If you only have:

yesterday's backup

and yesterday already contains the compromise, you’re in trouble.

Historical backups are extremely useful during incidents.

25. Test restoration before you need it

A backup that has never been restored is a theory.

At least occasionally:

restore
↓
start application
↓
check database
↓
check uploads
↓
check login

You don’t have to rebuild the entire production server every week.

But you should know the backup actually works.

26. Keep logs long enough to investigate

If logs disappear after a few days, forensic analysis becomes much harder.

At minimum, make sure you understand:

Apache access logs
Apache error logs
PHP logs
SSH logs
MariaDB logs
system logs
Virtualmin logs

and their rotation policy.

You don’t necessarily need enormous logs forever.

You need enough history to answer:

“What happened before the incident?”

27. Watch CPU after recovery

Because this incident involved a suspected cryptominer, CPU usage became one of the simplest things to watch.

For example:

top

or:

ps -eo pid,user,%cpu,%mem,etime,args \
--sort=-%cpu |
head -20

If a supposedly idle WordPress VPS suddenly spends:

300%
400%
600%

CPU on an unknown process, investigate.

Don’t immediately assume “miner.”

But investigate.

28. Check network connections too

For suspicious processes:

ss -tpn

You can look for established connections to unfamiliar remote addresses.

If you find:

unknown process
      ↓
unknown remote IP
      ↓
strange port

that’s worth investigating.

Again:

investigate first, kill second.

29. Firewall: default deny is your friend

I don’t need every port on the Internet.

For a typical web server, I might need:

22
80
443

plus Virtualmin/Webmin or other management ports as required.

I don’t need:

3306

publicly exposed just because one temporary application once needed it.

If I need remote database access, I’d rather use:

SSH tunnel
VPN
private network

than open MySQL globally.

30. Temporary access should actually be temporary

This sounds obvious.

But here’s what often happens:

Need MySQL
↓
open 3306
↓
create user@'%'
↓
finish project
↓
forget

Six months later:

3306 still open
user still exists
password never changed

That’s how temporary infrastructure becomes permanent attack surface.

If the task is temporary, create a temporary account and remove it afterward.

31. My simple post-incident firewall philosophy

I want the Internet to see:

HTTP
HTTPS
SSH

and only whatever management services I deliberately expose.

Everything else should have a reason.

For MySQL:

Internet
    X
    │
    └── 3306 blocked

SSH tunnel
    ↓
127.0.0.1:3306

That’s a much nicer architecture.

32. What I would do if I needed that temporary VPS tomorrow

I’d do:

Temporary VPS
       │
       │ SSH
       ▼
Main VPS
       │
       ▼
MariaDB localhost

On the temporary VPS:

ssh -N \
  -L 13306:127.0.0.1:3306 \
  user@main-server

Then configure the temporary application to use:

host = 127.0.0.1
port = 13306

When finished:

kill tunnel
destroy VPS

No MySQL firewall rule required.

No % account required.

No permanent public database endpoint.

33. One thing I would not do

I would not solve changing VPS IP addresses by doing:

0.0.0.0/0

and calling it a day.

That’s the easy solution.

It’s also the solution that makes me uncomfortable.

If a temporary server needs access, authenticate the temporary server rather than trusting the entire Internet.

34. The final security model

After everything was cleaned up, the model I wanted was roughly:

                         INTERNET
                             │
                 ┌───────────┴───────────┐
                 │                       │
                80                      443
                 │                       │
                 └──────────┬────────────┘
                            │
                         Apache
                            │
                         PHP-FPM
                            │
                       WordPress
                            │
                       MariaDB
                       localhost

SSH is separate:

Administrator
      │
      ▼
     SSH
      │
      ▼
    VPS

Temporary application server:

Temporary VPS
      │
      │ SSH tunnel
      ▼
Main VPS
      │
      ▼
127.0.0.1:3306

That’s considerably easier to reason about than:

Internet
   │
   ├── HTTP
   ├── HTTPS
   ├── SSH
   ├── MySQL
   ├── random management port
   └── forgotten application

35. The “before bed” checklist

If I’ve just recovered a server and I’m exhausted, this is the short version I’d keep beside the keyboard:

[ ] Backup verified
[ ] Suspicious files removed/quarantined
[ ] WordPress core verified
[ ] Plugins/themes checked
[ ] Unknown admin users removed
[ ] Unknown SSH keys removed
[ ] Cron checked
[ ] systemd checked
[ ] Unknown processes checked
[ ] Listening ports checked
[ ] MySQL users checked
[ ] MySQL remote access restricted
[ ] Firewall checked
[ ] SSH checked
[ ] Passwords rotated
[ ] API tokens rotated
[ ] Backups still working
[ ] Apache logs monitored
[ ] CPU/network monitored

If all of those are green, I can finally go to bed.

What this incident changed for me

Before this happened, I thought about server security mostly in terms of:

“Is everything patched?”

Afterward, I think about it more as:

“If one application gets compromised, how far can the attacker go?”

That’s a much better question.

Because vulnerabilities happen.

Plugins get abandoned.

Someone eventually installs something they shouldn’t.

A password eventually leaks.

A developer eventually makes a mistake.

The goal isn’t to build a server where nothing can ever go wrong.

The goal is to make sure that one mistake doesn’t automatically become total ownership of the machine.

The biggest lessons

If I had to keep only five things from this entire incident, they’d be these:

1. Keep historical backups

Not just the latest backup.

Historical backups give you a timeline.

2. Don’t trust filenames

A .jpg can be PHP.

A .png can be PHP.

A file called wp-env-setup.php doesn’t mean WordPress created it.

Use:

file

and inspect the contents.

3. Don’t expose services unnecessarily

Especially:

MySQL
Redis
Docker APIs
management interfaces

If you can keep them private, keep them private.

4. Least privilege applies everywhere

That means:

Linux users
WordPress users
database users
SSH keys
firewall rules
file permissions

5. When you have a trustworthy backup, don’t be afraid to restore

Sometimes the cleanest fix is not:

hunt every malicious line

It’s:

known-good backup
+
credential rotation
+
hardening
+
monitoring

Final thoughts

This wasn’t a Hollywood-style hack.

There was no glowing green terminal.

No mysterious guy in a hoodie.

Mostly there were:

grep
find
stat
file
tar
logs
coffee

and a lot of:

“Why the hell is this .png actually PHP?”

That’s what real server incidents tend to look like.

The useful part wasn’t identifying one clever attack.

It was slowly separating the evidence into three categories:

NORMAL
   ↓
SUSPICIOUS
   ↓
CONFIRMED

Once I stopped treating every strange line as proof of compromise, the investigation became much easier.

And once I had enough evidence to trust the historical backup, the recovery decision became straightforward.

Restore.

Rotate credentials.

Close unnecessary doors.

Limit privileges.

Monitor.

Then get some sleep.

That’s probably the most practical security advice I can give.

Don’t aim for a server that can never be compromised. Build one that is difficult to compromise, difficult to move around in, and easy to recover when something eventually goes wrong.

No Comments

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.