Hands up here, the outage of all sites was my fault. If you are interested, this post will look at what went wrong and how it was fixed.
Author Buzz is set up on a secure dedicated server. The domains are registered with a trustworthy registrar and will need to be renewed in another year or so. Everything is handled in-house and the only dev with root level access is me.
In theory, nothing can possibly go wrong.
Except, it did.
You see I had missed one tiny part of the chain – DNS. DNS is this fiddly thing that makes domain names and websites work. The specific DNS config for Author Buzz is what allows the WordPress setup to provide vanity web names like matthewdbrown.authorbuzz.co.uk or dev.authorbuzz.co.uk.
To keep things “clean” I used another domain name for the name server. The name server has worked flawlessly. However, I registered this other name a while ago. Long enough ago that my debit cards all expired and were replaced. So when the trustworthy registrar tried their best to renew the name server domain, guess what happened?
Yeah, you guessed it. The bank – quite rightly – refused to let that happen. Expired cards should not be used to facilitate payments. That is why cards expire in the first place.
In many post-disaster reviews, this section might have a lot of technical details. However, the fix was really simple – I whipped out my card and paid for the renewal.
That gives me a full year to figure out if I can set things up with slightly fewer moving parts. It was a silly mistake and I feel very silly for having let it happen.
This outage taught me one thing – you only need to overlook one thing for everything to come crashing down for a while. If anything this has increased the respect I feel for the devs that run Twitter and generally keep everything online. Likewise, the mavericks that innovate online technology businesses.
I am definitely open to partnering up with others to deliver our vision of a launching pad for UK writers. If only because with more of us on the project, there are more of us ready to catch silly mistakes like that one.
And on that note, I think it is time for me to stop beating myself up and go and put the kettle on.
Domain renewals are easy to overlook, did you not get warning emails prior to expiry? I don’t think this will prevent this kind of downtime, but I’m willing to set up secondary nameservers for your domains on my servers. In all honesty (looking at the linked site), I’m not sure why your DNS is working at all
https://intodns.com/authorbuzz.co.uk
I’ve had years of experience setting this stuff up (not using control panels) if you want me to look at anything
I wouldn’t say no to a second pair of eyes on things, Danny. It does look like I could speed up the NS-A resolution by a step. Good catch.
