Sites outage – confession of a silly dev

train crash
Bugs
2

Hands up here, the outage of all sites was my fault. If you are interested, this post will look at what went wrong and how it was fixed.

How the outage happened

Author Buzz is set up on a secure dedicated server. The domains are registered with a trustworthy registrar and will need to be renewed in another year or so. Everything is handled in-house and the only dev with root level access is me.

In theory, nothing can possibly go wrong.

Except, it did.

You see I had missed one tiny part of the chain – DNS. DNS is this fiddly thing that makes domain names and websites work. The specific DNS config for Author Buzz is what allows the WordPress setup to provide vanity web names like matthewdbrown.authorbuzz.co.uk or dev.authorbuzz.co.uk.

To keep things “clean” I used another domain name for the name server. The name server has worked flawlessly. However, I registered this other name a while ago. Long enough ago that my debit cards all expired and were replaced. So when the trustworthy registrar tried their best to renew the name server domain, guess what happened?

Yeah, you guessed it. The bank – quite rightly – refused to let that happen. Expired cards should not be used to facilitate payments. That is why cards expire in the first place.

How I fixed the outage

In many post-disaster reviews, this section might have a lot of technical details. However, the fix was really simple – I whipped out my card and paid for the renewal.

That gives me a full year to figure out if I can set things up with slightly fewer moving parts. It was a silly mistake and I feel very silly for having let it happen.

What I learned from the outage

This outage taught me one thing – you only need to overlook one thing for everything to come crashing down for a while. If anything this has increased the respect I feel for the devs that run Twitter and generally keep everything online. Likewise, the mavericks that innovate online technology businesses.

I am definitely open to partnering up with others to deliver our vision of a launching pad for UK writers. If only because with more of us on the project, there are more of us ready to catch silly mistakes like that one.

And on that note, I think it is time for me to stop beating myself up and go and put the kettle on.

Domain renewals are easy to overlook, did you not get warning emails prior to expiry? I don’t think this will prevent this kind of downtime, but I’m willing to set up secondary nameservers for your domains on my servers. In all honesty (looking at the linked site), I’m not sure why your DNS is working at all

https://intodns.com/authorbuzz.co.uk

I’ve had years of experience setting this stuff up (not using control panels) if you want me to look at anything

2 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

no spam
Bugs
1
A few potential bug fixes and spam defences

I’ve been looking into two seemingly unrelated bugs that I now suspect share a root cause. This evening, I added some new directives that control cache expiry. In both cases, this is for JSON content which is what the cache shows you when you want to see a post and …

Bugs
1
CSS error on the front page

Something has gone screwy with the CSS for the front page. I’m already working on fixing it.

Bugs
Rare JSON/HTML bug

I have been made aware of a rare bug that may only impact Safari users. The exact cause is unknown but most likely cache-related. It can cause JSON to be served up instead of HTML for new posts. I believe that 99% of users will most likely never see the …

Skip to toolbar