More Control Over Your Server

If you work with servers long enough, you naturally start collecting tools that make life easier. For me, those were Webmin and Virtualmin for years. Not because there was no other way, but simply because it's efficient: less thinking about side issues and more focus on what you actually want to run.

However, there is a flip side to that. The more that is arranged for you, the less you actually see of what is happening. And at some point, that started to bother me. Not because it didn't work, but precisely because it went too smoothly.

So this time I decided to do things differently. Just doing it myself again. Writing configurations, making choices, breaking things, and then figuring out why.

The basics remain the basics ​

As always, you start with the familiar things: locking down SSH, using keys, and installing Fail2Ban. That's not exciting, but it is necessary. A simple jail for SSH looks like this, for example:

[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 5
bantime = 3600

You basically never look at that again. Until you open your logs.

What is really happening ​

At some point, I started looking more closely at my Apache logs, just out of curiosity. Then you suddenly see how much garbage passes by. Requests to /phpmyadmin, /wp-login.php, .env files, and old exploits. It is not targeted and not particularly smart, but it doesn't stop either.

The tricky part is that technically there is nothing wrong with those requests. They are just valid HTTP calls, so Fail2Ban does nothing with them. And that is the point where you realize that "it works" is not the same as "it's okay".

Adding ModSecurity in between ​

That's why I started using ModSecurity, not as a replacement but as an extra layer. In Apache, the basics are fairly simple:

<IfModule security2_module>
    SecRuleEngine On
    SecRequestBodyAccess On

    SecAuditEngine RelevantOnly
    SecAuditLog /var/log/apache2/modsec_audit.log

    IncludeOptional /etc/modsecurity/crs/crs-setup.conf
    IncludeOptional /etc/modsecurity/crs/rules/*.conf
</IfModule>

With the OWASP ruleset included, it immediately starts doing something. Requests that initially just passed through Apache and got a 404 are now blocked. Think of SQL injection attempts, weird headers, and known paths being abused. Your application doesn't even see them anymore, and your logs become much clearer.

The standard rules are quite strict, though. You especially notice this if you build something yourself or have an API. Sometimes you have to disable something:

SecRuleRemoveById 941100

Or just for a specific path:

<Location "/api/">
    SecRuleRemoveById 942100
</Location>

Not complicated, but something to keep in mind.

And if something does get through ​

Up to this point, everything is on the frontend, and you try to keep bad requests out. That works well, but says nothing about what happens if something succeeds or enters through another route. That's why I added AIDE.

AIDE ​

AIDE doesn't look at traffic, but at your system itself. It takes a snapshot and compares it later. Initialize first:

aideinit
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

Afterward, you can check:

aide --check

In the config, you indicate what is important:

/bin    NORMAL
/sbin   NORMAL
/etc    NORMAL

It's fairly boring stuff, until something changes that you didn't expect.

Putting it all together ​

What you ultimately end up with isn't a single solution, but a combination. Fail2Ban tackles obvious abuse, ModSecurity filters requests before they do anything, and AIDE checks if your system is still correct. On their own, they don't amount to much, but together they just provide more control.

Finally ​

The biggest difference isn't even in the tools, but in what you see. As soon as you take your logs seriously, your perspective naturally changes. Usually, that is also the moment when you start doing just a little more than merely thinking everything is fine.