The most useful thing in an incident isn’t a scary-looking file. It’s the sequence of events.
By SepedaTua — CrushEdge.com
After I had enough evidence to know which files were malicious, I wanted to answer the question that had been bothering me since the beginning:
When did the attacker actually get in?
This sounds simple.
It isn’t.
A compromised server can contain timestamps from several different events:
- when a file was created,
- when it was modified,
- when its metadata changed,
- when somebody accessed it,
- when a backup captured it,
- and when the web server logged a request.
Those are not necessarily the same moment.
So I stopped trying to find one magical timestamp.
Instead, I started building a timeline.
1. The four timestamps I care about
Linux gives us several useful timestamps.
For a file:
stat suspicious.php
you may see:
Access
Modify
Change
Birth
They mean different things.
Birth
When the filesystem says the file was created.
Modify
When the file contents were last modified.
Change
When filesystem metadata changed.
Access
When the file was last accessed, assuming access-time updates are being recorded.
For incident response, Birth, Modify, and Change are generally much more interesting than Access.
2. But timestamps are evidence, not proof
This is important enough to repeat.
Suppose I see:
Birth: 2026-08-19 06:35:39
Modify: 2026-08-19 06:35:40
Change: 2026-08-20 03:52:07
I shouldn’t automatically conclude:
“The attacker uploaded it at exactly 06:35:39.”
The filesystem timestamp tells me when the file appeared according to the filesystem.
It doesn’t tell me exactly how it appeared.
Maybe PHP created it.
Maybe a restore process created it.
Maybe somebody copied it.
Maybe a script changed it.
I need other evidence.
3. The web access log is one of those pieces
The access log tells me what the web server received.
For example:
19/Aug/2026:03:50:...
POST /something
That’s useful.
But again, the absence of a log entry doesn’t necessarily prove something didn’t happen.
Logs can rotate.
Logs can be incomplete.
The request may have gone through another service.
The attacker may have used another access path.
So I treat logs as another timeline layer.
4. One of my early mistakes was searching the wrong time
During the investigation, I initially concentrated on a narrow window such as:
03:40–04:00 UTC
and searched the access log for:
POST
wp-admin
wp-login
admin-ajax
upload
xmlrpc
plugin
The result was:
0 lines
At first glance, that was reassuring.
But it wasn’t enough.
The suspicious files had timestamps that didn’t line up with that particular window.
That’s an important lesson:
Don’t make your investigation depend on a timestamp you haven’t independently established.
5. Start with the suspicious file
If I have:
bf6f03.php
I first record:
stat /home/customer/public_html/bf6f03.php
If the file has already been removed, I use the preserved evidence copy if I made one.
Then record its hash:
sha256sum /root/evidence/bf6f03.php
Now I have a stable reference:
filename
size
timestamps
owner
permissions
hash
That’s much better than:
“There was some weird PHP file.”
6. Then search for files created around the same time
For example:
find /home/customer/public_html \
-type f \
-newermt '2026-08-19 06:30:00 UTC' \
! -newermt '2026-08-19 06:40:00 UTC' \
-printf '%TY-%Tm-%Td %TH:%TM:%TS | %u:%g | %m | %s | %p\n' \
2>/dev/null |
sort
Now I’m looking for a cluster.
If one suspicious file appears, that’s interesting.
If twenty suspicious files appear within the same minute, that’s much more interesting.
7. Clusters are valuable
Imagine I find:
06:35:39 random1.php
06:35:39 random2.php
06:35:40 strange.jpg
06:35:40 random3.php
06:35:41 random4.gif
That suggests something happened around that period.
Maybe one exploit dropped multiple files.
Maybe a loader deployed a payload.
Maybe a compromised plugin created them.
I still don’t know which.
But now I know where to concentrate.
8. Look for directory changes too
Files aren’t the only useful evidence.
Check the suspicious directory:
stat /home/customer/public_html/suspicious-directory
A directory’s metadata can sometimes help establish when files were added.
For example:
Modify: 2026-08-19 06:35:41
combined with several files created around that time is useful corroborating evidence.
Again:
not proof by itself.
9. Compare against the backup
This became one of the strongest pieces of evidence in my investigation.
Suppose:
known-good backup
│
└── suspicious.php absent
current filesystem
│
└── suspicious.php present
That tells me the file appeared sometime between those states.
If the backup was taken on August 17 and the suspicious file first appears on August 19:
Aug 17
│
│ clean backup
│
▼
Aug 19
│
│ suspicious file
▼
Current
Now my search window becomes much smaller.
10. Differential backups need special care
This was another place where I initially had to slow down.
A Virtualmin backup may be:
full
or:
differential
or otherwise structured according to Virtualmin’s backup system.
A differential archive isn’t necessarily a standalone copy of the entire website.
So if:
tar -tzf some-backup.tar.gz
doesn’t show a file, don’t immediately conclude:
“That file definitely didn’t exist.”
First understand what that archive represents.
11. The full backup gave me the proper baseline
The full Virtualmin backup showed the expected structure:
./public_html/
./wp-admin/
./wp-content/
./homes/
./Maildir/
That was much more useful for understanding the account layout.
Once I understood that, the backup investigation became much less confusing.
12. File creation time and backup time create a boundary
Suppose:
Backup:
August 17, 00:00
Suspicious file:
August 19, 06:35
The only conclusion I can safely make is:
The file wasn’t part of that backup archive, and it existed on the later filesystem.
That gives me a boundary.
It does not tell me the exact exploit time.
That’s an important distinction for a blog post like this.
I don’t want to invent a precise attack timeline just because a timestamp looks convenient.
13. Then correlate with access logs
Once I have a reasonable time range, I search the web logs.
For example:
grep -E \
'\[19/Aug/2026:06:(3[0-9]|4[0-9]):' \
/path/to/access_log
Then narrow the interesting requests:
grep -Ei \
'POST|wp-admin|wp-login|admin-ajax|upload|install|plugin|xmlrpc|rest|ajax' \
But I don’t stop there.
I also look at the raw surrounding lines.
Context matters.
14. Don’t search only for the suspicious filename
Suppose the malicious file is:
bf6f03.php
It is tempting to search:
grep 'bf6f03.php' access_log
That’s useful if the attacker directly requested it.
But the initial compromise may have used:
/wp-admin/
/wp-json/
/wp-admin/admin-ajax.php
/wp-login.php
or a vulnerable plugin endpoint.
The malicious file may have been created by a request that never mentioned its final filename.
15. Look for POST requests
During an incident, POST requests deserve attention.
For example:
grep -E '"POST ' /path/to/access_log
Then narrow the time range.
I’m particularly interested in POST requests involving:
wp-admin
admin-ajax.php
wp-json
upload
plugin
theme
install
But again:
POST does not mean attack.
WordPress legitimately uses POST everywhere.
16. Look for successful responses
A suspicious request returning:
404
is very different from:
200
Likewise:
403
means the server rejected something.
So when reviewing the log, I want to see:
timestamp
IP
method
URI
status
response size
user-agent
referrer
That gives me context.
17. The response size can be surprisingly useful
Imagine:
POST /something 404 512
versus:
POST /something 200 8432
The second deserves much more attention.
Not because 200 proves compromise.
It doesn’t.
But it tells me the application successfully handled the request.
18. Cloudflare makes IP analysis more interesting
A lot of the traffic in the logs came through Cloudflare addresses.
That means the source IP visible at the web server isn’t necessarily the attacker’s actual public IP.
This is a common mistake:
“I found the IP in Apache logs, therefore I found the attacker.”
Not necessarily.
If a reverse proxy/CDN sits in front of the server, you need to understand the proxy headers and logging configuration.
Otherwise you’re looking at the intermediary.
19. But attacker infrastructure can still be useful
Even when an IP is behind a proxy or cloud provider, other details can be useful:
user-agent
request path
timing
request frequency
referrer
HTTP method
payload behavior
repeated requests
The pattern is often more useful than the IP.
20. Don’t waste hours trying to identify the human
This was another lesson.
It’s tempting to think:
“I found an IP. Who is this person?”
Maybe you can identify:
hosting provider
ASN
country
cloud provider
reverse DNS
That’s sometimes useful.
But for practical recovery, the more important question is:
“What did this connection do?”
Knowing that an IP belongs to a VPS provider is less useful than knowing it successfully triggered a vulnerable endpoint.
21. The attacker may not be one person
There may be:
scanner
initial exploit
payload server
webshell operator
automated bot
secondary malware
all involved.
So I prefer the term:
attacker infrastructure
rather than pretending I know exactly who sat behind the keyboard.
22. Build a timeline table
This is something I’d recommend to anyone investigating a similar incident.
Create a simple table:
| Time | Evidence | Confidence |
|---|---|---|
| Aug 17 | Known-good backup | High |
| Aug 19 06:35 | Suspicious file created | High |
| Aug 19 | Suspicious files appear | High |
| Aug 19 | Relevant web requests | Medium/High |
| Aug 20 | Malicious file discovered | High |
| Aug 20 | Account quarantined | High |
| Aug 20 | Restore performed | High |
The confidence column is important.
It stops me from accidentally turning assumptions into facts.
23. Separate facts from theories
For example:
Fact
bf6f03.php existed on the compromised filesystem.
Fact
bf6f03.php was not present in the earlier backup examined.
Fact
The file contained an obfuscated PHP execution mechanism.
Theory
An attacker uploaded bf6f03.php through a WordPress vulnerability.
The theory may be extremely plausible.
But unless I have the request that caused the upload, I shouldn’t present it as proven.
This distinction makes an incident report much more trustworthy.
24. The same applies to the entry point
If I don’t have evidence showing:
vulnerable-plugin-endpoint
↓
file upload
↓
webshell
then I shouldn’t write:
“The attacker exploited Plugin X.”
I can write:
“The evidence is consistent with exploitation through Plugin X, but the exact initial entry point could not be confirmed.”
That’s honest.
25. Why this matters to readers
A security blog can easily become a collection of scary claims:
“Hackers used advanced techniques!”
“The attacker breached the server!”
“This vulnerability allowed complete control!”
That sounds exciting.
It isn’t very useful.
I’d rather tell readers:
Here is what I observed.
Here is what I can prove.
Here is what I suspect.
Here is what I don't know.
Here is what I did.
That’s much more useful to another sysadmin at 1 AM.
26. The final timeline doesn’t need to be perfect
I don’t need:
06:35:39.719723167
attacker pressed button
That’s fake precision.
What I need is:
Before Aug 17:
known-good state
Between Aug 17 and Aug 19:
compromise occurred
Aug 19:
malicious files appeared
Aug 20:
I discovered the compromise
Aug 20:
affected accounts restored/quarantined
After restoration:
verification and hardening
That’s enough to tell the story accurately.
27. And this changed my investigation style
At the beginning, my approach was:
find suspicious thing
↓
grep around
↓
find another suspicious thing
↓
grep more
That gets messy quickly.
The better approach became:
known-good point
↓
known-bad point
↓
identify differences
↓
build timeline
↓
correlate logs
↓
classify evidence
↓
determine scope
Much cleaner.
28. The most useful question wasn’t “Who hacked me?”
It was:
“What can I prove happened between the last known-good state and the first known-bad state?”
That question kept me from going down several rabbit holes.
It also gave me a practical stopping point.
Once I could establish:
known-good backup
+
known malicious files
+
affected accounts
+
reasonable timeline
+
recovery
I had enough information to make the server safe again.
29. There will always be something else to investigate
You can spend forever looking at:
Apache logs
MariaDB logs
SSH logs
PHP logs
WordPress logs
Cloudflare logs
cron
systemd
filesystem timestamps
DNS
IP addresses
user agents
Eventually you have to decide:
Is this information going to change what I do?
If the answer is no, stop.
That’s one of the hardest lessons in incident response.
30. The practical rule I now use
I ask:
Does this evidence change the scope?
If yes:
investigate.
Does it change the recovery plan?
If yes:
investigate.
Does it identify another persistence mechanism?
If yes:
investigate immediately.
Is it merely interesting?
Put it in the notes and move on.
What I ultimately wanted from the investigation
Not a CSI-style reconstruction.
Not the attacker’s life story.
Not a perfect explanation of every request.
I wanted five answers:
1. Which accounts were affected?
2. Which files were malicious?
3. When did they appear?
4. Was there a clean backup?
5. Can I restore and prevent recurrence?
Once I could answer those, the investigation had done its job.
Everything after that was useful only if it improved the security of the server.
And that’s probably the biggest practical lesson I got from this whole mess:
Incident response is not about knowing everything. It’s about knowing enough to make the right decision.
No Comments