For any website manager, one of the most stressful moments in the job comes when the website gets hacked into. Even more so for smaller websites that don’t have a team of experts who look after such things.

A website like this one for example, which suffered a hack last Thursday.

It was an unusual attack, in that it didn’t try to delete the site or post “oh so funny” threats from Anonymous, which seemed to be all the rage a few years back. This attack actually went to a lot of effort to stop me taking the site down for maintenance, because their plan was to flood the pages with adverts, and earn an income from my distress.

I’ve flagged the fraudster to Google, as it’s their AdSense code being used in this way. Google wont tell me what will happen, but at a best they’ll close the fraudulent account without paying anything.

Of course, I am out of pocket as well.

But, it’s the helpless frustration that comes from playing whack-a-mole with the hacker as you desperately try to find how they got in, and stop them screwing the website, while it is still live to visitors who are now complaining about the nasty adverts on there, that’s so stressful.

In the olden days of the internet, in the halcyon times of a couple of years ago, there was a lot of hacking going on, but most of it was “drive by shootings” by simple scripts usually downloaded off the internet by spotty faced kids who sat in bedrooms thinking they were being terrifically clever by using code someone else had written.

A lot of that dried up as websites and software got faster at fixing bugs when they were found.

These days hacking is harder, and hence it’s usually the clever people doing it. Which sadly also means that unlike the old days when fixing the problem was a simple task, these days it’s much harder to find the flaw and repair it.  It also means that with a lot more effort needed to find flaws, that the hacker invests a lot more work in making it a damn nuisance to repair.

In this situation, a series of template files were being changed to add the annoying adverts to the website, and if I deleted the files, within a minute or two, the entire folder would reappear. Renaming the folder and changing how the website architecture worked, to give me some breathing space to fix things properly lasted, oh, about 5 minutes.

It became apparent that the hacker had a script that was able to find template references in other files, then attack the templates, no matter where they were.

It also became apparent quickly on that there was a copy of the template files somewhere on the server that the hacker had made, so they could override any changes I was making.

So far, this looked like an annoying, but effective “bot” attack.

What was puzzling was that changing the entire website architecture didn’t stop the attack. If they were pinging a file maliciously inserted into the website to launch the attacks, then renaming every folder on the server should have stopped them, at least until they found the new folder name.

In the end I decided to turn the website off by amending a few core files to point everything to a maintenance page, and got to work migrating the website to a new server, which would hopefully be clean enough to get the site working again.

Then it got interesting

Those core files, edited in a way that was obvious to a human, but not a computer, were being edited back again a while later. Even changing file names subtly saw them being renamed back a while later.

While the attack looked like a computer bot, and the speed of its responses suggested as much, this new development was clearly the actions of a human who had access to the website and was now fighting my attempts to turn the website off.

After all, if I did that then they’d lose a day or two of fraudulent income.

What had been me fighting a bit of computer code maliciously inserted into the website and causing havoc was now a fight between two humans — one seemingly in Albania and the other here in London. My moves to shut-down the website being undone a hour or so later by my adversary.

This made the fight more personal. It’s one thing to be dealing with a computer code being mean, but when you can see that an actual human is editing your website as you watch, then it’s a more tangible experience. And a much more annoying one.

While working to migrate all the clean code to a new webserver, I continued to work on trying to fix the old, and finally, it looked like a change I made resulted in silence from the hacker. This was interesting, changing a folder which was in a location that wasn’t accessible externally to be unavailable to the server as well by altering its permissions, blocked the hacker.

Watching for a few hours, and calm reigned

I am wary of declaring victory, but after a couple of hours of watching, and no more attacks, then it was time to tentatively decide to switch the website on again.

The folder was in a non-web addressable location, which is to say, the webserver could access it, but no one else could. This is a modest security precaution, as it’s where uploaded images for the events listing would be saved. Making sure that only images could be uploaded, and putting them somewhere that makes it harder for a malicious payload to be triggered is a routine thing to do.

Somehow, the hacker had uploaded a malicious payload disguised as an image to the folder, and it was merrily playing havoc with the website.

But most frustrating is knowing that it’s my own fault that the hacker got into the website. I wrote that bit of code, and took all the precautions I know to take, but it was evidently not good enough.

I shouldn’t beat myself up too much, as the biggest cleverest of websites get hacked as well, and I am never going to be able to stop a determined attack — but it still rankles that at some level, it was my code that created the vulnerability.

A little over a million lines of code are on this website, and it takes just one glitch in there to allow a hacker inside.

Repairing a hack like this takes a lot of effort, just to find the cause, let alone find a fix.

I was recently involved in a big problem at work, and described as “cold as ice” in how I responded… which is to say here is a problem, let’s find a solution.

It’s how I look externally, while inside my stomach is churning like mad, because when a website is being hacked, as much as you really want to, running around wailing and crying wont change the result. Be methodically emotionless, fix the problem and don’t let anyone see your stomach making you feel sick and the massive levels of stress you pile on yourself.

When it’s fixed (hopefully), then the stresses boil over and you curl up in a corner and cry.

Fortunately websites being hacked is a decreasing problem

Just a few years ago, I doubt I went more than six months without a serious hack on a website, such were the quantity of holes in code back then. Over the years, websites have got better at finding exploits and fixing them.

I am still not entirely sure how to enable the functionality I want, without opening that door once more to a risk of the website being hacked again, so for a while you’ll see a lot of missing images on the website while I ponder more calmly how to deploy a long-term solution.

I am still wary of declaring this attack to be over, but if it is indeed finished, then with luck, it’ll be another couple of years before the website is hacked again.

But one day… it’ll happen again.