Part 3 — I Needed Remote MySQL Access, So I Made It Too Easy

Part 3 of a real-world WordPress VPS compromise investigation

By SepedaTua — CrushEdge.com


There was another problem waiting for me after the filesystem investigation.

MySQL.

And this one was partly my own fault.

I had a legitimate reason for allowing remote MySQL connections.

Some of my applications live on temporary cloud VPS instances. I create a VPS when I need it, run the application, and eventually destroy it.

Because those servers are temporary, their public IP addresses aren’t something I want to hard-code into a permanent firewall rule.

So I had taken the easy route.

And “easy” is often where server security starts getting interesting.


Why I needed remote MySQL

The setup was roughly this:

Temporary application VPS
          │
          │ MySQL connection
          ▼
   Main database server

The temporary VPS might exist for:

  • testing
  • running a temporary application
  • migration work
  • a short-lived API
  • data processing
  • development

Then I destroy it.

Later I might create another VPS.

The new VPS gets another public IP.

So the obvious solution:

ALLOW 203.0.113.42

doesn’t work very well for me.

The next time I create a VPS, the IP might be:

203.0.113.87

Then I’d have to modify the firewall and MySQL permissions again.

That’s manageable.

But it’s annoying.

And, as I’ve learned many times while maintaining servers, annoying configuration has a habit of eventually becoming dangerous configuration.


The dangerous shortcut

The shortcut is something like:

user@'%'

or a MySQL server listening on:

0.0.0.0

with a user that has substantial privileges.

This effectively says:

“This database account may connect from anywhere.”

That’s a very different security model from:

“This particular application server may connect to this particular database.”

The first one dramatically increases the attack surface.


0.0.0.0 isn’t automatically the vulnerability

This is worth explaining carefully.

Seeing:

bind-address = 0.0.0.0

doesn’t automatically mean your database has been compromised.

It means MySQL is listening on all available network interfaces.

Whether that is actually reachable from the Internet depends on other controls:

  • firewall
  • cloud security groups
  • routing
  • network ACLs
  • TCP port exposure
  • MySQL authentication
  • account host restrictions

So don’t see 0.0.0.0 and immediately panic.

But if MySQL is Internet-reachable and the accounts accept connections from arbitrary hosts, that’s a very different situation.


The MySQL account that worried me

The configuration effectively allowed a remote database account to connect broadly.

Conceptually:

database_user@'%'

The % is the important part.

It means:

any host

not:

my temporary VPS

That’s convenient.

It’s also exactly the kind of convenience I don’t want on an Internet-facing production database server.


Why I didn’t simply remove remote MySQL

Because that would break something I actually needed.

This is an important point when securing real systems.

Security advice sometimes sounds like:

“Just disable remote access.”

Sure.

If you don’t need remote access, that’s exactly what I’d recommend.

But I did need it.

So the real question was:

How can I keep the functionality without leaving MySQL open to the whole Internet?

That’s a much better question than simply asking how to close the port.


Solution #1: VPN

The cleanest long-term design is to stop exposing MySQL directly to the Internet.

Instead:

Temporary VPS
      │
      │ encrypted VPN
      ▼
Private network
      │
      ▼
MySQL

For example, a VPN such as WireGuard can give the temporary VPS a stable private address even though its public IP changes.

The database can then be restricted to the VPN network.

Conceptually:

10.50.0.0/24

and MySQL can allow connections from that private network rather than:

%

This is considerably cleaner.


But there’s a catch with temporary VPSs

The temporary VPS still needs to join the VPN.

That’s not difficult, but it adds another step to the lifecycle:

create VPS
   ↓
install application
   ↓
install/configure VPN
   ↓
connect to database
   ↓
destroy VPS

For a VPS that exists for five minutes, that may feel like overkill.

For a system that accesses production data, I think it’s worth considering.


Solution #2: SSH tunnel

For occasional administrative access, an SSH tunnel is often simpler.

Instead of:

Internet
   │
   ▼
MySQL :3306

you can use:

Temporary VPS
     │
     │ SSH
     ▼
Main server
     │
     ▼
MySQL localhost:3306

The application connects to:

127.0.0.1:3306

on the temporary VPS, while SSH forwards the traffic through the server.

For a temporary environment, this can be very convenient.

The downside is that the application has to maintain the SSH tunnel.


Solution #3: Firewall restrictions

If you absolutely need direct TCP access to MySQL, firewall rules are still useful.

But here’s the problem I originally had:

The temporary VPS IP changes.

That doesn’t mean the answer has to be:

ALLOW everyone

You can instead automate the lifecycle.

For example:

Create VPS
   ↓
discover its public IP
   ↓
add temporary firewall rule
   ↓
run application
   ↓
destroy VPS
   ↓
remove firewall rule

The important word is temporary.

The firewall rule should live only as long as the application needs it.


Solution #4: A private cloud network

If your VPS provider supports private networking, that’s another good option.

The public Internet then becomes irrelevant to the database connection.

Conceptually:

                 Internet
                    │
             ┌──────┴──────┐
             │             │
        Web traffic     SSH/admin
             │
             ▼
        Application VPS
             │
             │ private network
             ▼
        Database server

That’s the architecture I’d prefer for infrastructure I expect to keep around.


What I would NOT do now

I would not solve the problem by creating:

'user'@'%'

with:

ALL PRIVILEGES

and then assuming a strong password makes everything fine.

A strong password is good.

It is not a substitute for network restriction.

If an attacker can reach the service, you’ve increased the number of things that need to be correct:

network exposure
+
authentication
+
password
+
privileges
+
software vulnerabilities
+
configuration

I’d rather remove unnecessary exposure entirely.


Least privilege matters too

There is another mistake that’s easy to make.

Suppose an application only needs to run:

SELECT
INSERT
UPDATE

Why give it:

DROP
ALTER
CREATE
GRANT OPTION

too?

It doesn’t need them.

A compromised application becomes much more dangerous when its database credentials have unnecessary privileges.

So the account should ideally have only what the application actually needs.

For example, conceptually:

GRANT SELECT, INSERT, UPDATE
ON app_database.*
TO 'app_user'@'some-restricted-host';

The exact privileges depend on what the application actually does.


And don’t forget the database itself

Another mistake is restricting the MySQL user but leaving port 3306 publicly reachable.

That’s still unnecessary exposure.

If only a VPN network needs access, the firewall should reflect that.

If only a small set of machines needs access, the firewall should reflect that.

The ideal architecture has multiple layers:

Internet
   │
   X 3306 blocked
   │
Firewall
   │
   ▼
Allowed network
   │
   ▼
MySQL
   │
   ▼
Restricted database user

If one layer fails, another layer still provides some protection.


During the incident, I changed my priorities

Before the compromise, my thinking was:

“I need remote MySQL, so I’ll make it reachable.”

After the compromise, my thinking became:

“I need remote MySQL, but why should every random Internet host even be able to knock on the door?”

That’s a much healthier question.


The temporary VPS problem has a good answer

For my particular use case, I don’t actually need the temporary VPS to have a permanent public IP.

I need it to have a stable identity on the private network.

That’s the key distinction.

Instead of identifying the machine by:

public IP

identify it by something controlled inside the private network:

VPN identity

or:

private IP

or:

SSH key

depending on the architecture.

Then the public IP can change without requiring MySQL to become globally accessible.


One more thing: rotate credentials after a compromise

This is easy to forget.

If I believe the server was compromised, I don’t want to assume that existing database credentials remained secret.

The same goes for:

  • WordPress administrator passwords
  • FTP/SFTP credentials
  • SSH credentials
  • API keys
  • application secrets
  • SMTP passwords
  • database passwords
  • deployment keys

Even if I don’t find evidence that a particular credential was stolen, the safer assumption after a serious compromise is:

If the attacker could have read it, rotate it.

That’s particularly important for credentials stored in files on the compromised server.


My recovery checklist

By this stage, my recovery checklist looked more like this:

[ ] Restore from a known-good backup
[ ] Remove confirmed malicious files
[ ] Search for known indicators of compromise
[ ] Check disguised image files
[ ] Check suspicious PHP files
[ ] Check executable files
[ ] Check cron jobs
[ ] Check systemd services
[ ] Check SSH authorized_keys
[ ] Check WordPress administrators
[ ] Rotate important credentials
[ ] Restrict MySQL network access
[ ] Reduce MySQL privileges
[ ] Update WordPress/plugins/themes
[ ] Update the operating system
[ ] Start MariaDB
[ ] Verify database access
[ ] Start Apache
[ ] Monitor logs

Notice what’s missing:

[ ] Delete one suspicious PHP file
[ ] Declare victory

That’s not enough.


What I learned from the MySQL side

The lesson wasn’t:

“Never allow remote MySQL.”

Sometimes remote database access is perfectly legitimate.

The lesson was:

Don’t solve a temporary networking problem by creating permanent global exposure.

If a temporary VPS needs database access, give it a temporary, controlled path.

VPN.

SSH tunnel.

Private networking.

Temporary firewall rules.

There are several options.

Just don’t turn:

temporary VPS

into:

Internet-wide database client

because changing IP addresses is inconvenient.


The uncomfortable part

There was still one question I couldn’t answer from the filesystem alone:

How did the attacker initially get in?

I had plenty of evidence showing what they did after gaining access.

But post-compromise artifacts don’t necessarily tell you the original entry point.

It could have been:

  • a vulnerable plugin
  • stolen WordPress credentials
  • a vulnerable application
  • an exposed service
  • a previously compromised account
  • another website on the same server
  • or something I hadn’t identified yet

I didn’t want to invent a story just because one explanation sounded plausible.

So I went back to the logs.

And that’s where the investigation became much more interesting.


Coming next

Part 4 — The Log Lines That Weren’t There

The strange part of this incident was not only what appeared in the access logs.

It was also what didn’t appear.

I’ll go through the timeline, the missing requests, Cloudflare/proxy effects, why filesystem timestamps didn’t line up neatly with HTTP logs, and why a clean-looking access log does not prove that a webshell wasn’t used.

No Comments

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.