Sites outage – confession of a silly dev

train crash
Bugs
1

Hands up here, the outage of all sites was my fault. If you are interested, this post will look at what went wrong and how it was fixed.

How the outage happened

Author Buzz is set up on a secure dedicated server. The domains are registered with a trustworthy registrar and will need to be renewed in another year or so. Everything is handled in-house and the only dev with root level access is me.

In theory, nothing can possibly go wrong.

Except, it did.

You see I had missed one tiny part of the chain – DNS. DNS is this fiddly thing that makes domain names and websites work. The specific DNS config for Author Buzz is what allows the WordPress setup to provide vanity web names like matthewdbrown.authorbuzz.co.uk or dev.authorbuzz.co.uk.

To keep things “clean” I used another domain name for the name server. The name server has worked flawlessly. However, I registered this other name a while ago. Long enough ago that my debit cards all expired and were replaced. So when the trustworthy registrar tried their best to renew the name server domain, guess what happened?

Yeah, you guessed it. The bank – quite rightly – refused to let that happen. Expired cards should not be used to facilitate payments. That is why cards expire in the first place.

How I fixed the outage

In many post-disaster reviews, this section might have a lot of technical details. However, the fix was really simple – I whipped out my card and paid for the renewal.

That gives me a full year to figure out if I can set things up with slightly fewer moving parts. It was a silly mistake and I feel very silly for having let it happen.

What I learned from the outage

This outage taught me one thing Рyou only need to overlook one thing for everything to come crashing down for a while. If anything this has increased the respect I feel for the devs that run Twitter and generally keep everything online. Likewise, the mavericks that innovate online technology businesses.

I am definitely open to partnering up with others to deliver our vision of a launching pad for UK writers. If only because with more of us on the project, there are more of us ready to catch silly mistakes like that one.

And on that note, I think it is time for me to stop beating myself up and go and put the kettle on.

Domain renewals are easy to overlook, did you not get warning emails prior to expiry? I don’t think this will prevent this kind of downtime, but I’m willing to set up secondary nameservers for your domains on my servers. In all honesty (looking at the linked site), I’m not sure why your DNS is working at all

https://intodns.com/authorbuzz.co.uk

I’ve had years of experience setting this stuff up (not using control panels) if you want me to look at anything

1 Comment

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

train crash
Bugs
2
Tiny MCE and BBPress bug

I guess I spoke too soon. The update seems to have had a very strange side effect – BBPress’ WYSIWYG editor has stopped working. I noticed a bug report from one of our members this evening. They were reporting that the forums were being squiffy. I was going to reply …

Bugs
No bug found (but some UI improvements would help)

Time for a developer ramble about the state of things here at Author Buzz UK, bug tracking, progress, and article length. Solving Bugs We reported that a potential bug had been found with sign-ups. Thanks to some truly amazing beta testers we have found that there probably is no bug …

Bugs
3
Oh no! Not now!

Right in the middle of NaNoWriMo, the server got the hiccups. First, the blogs stopped responding. Then the main site went down. I did what any stressed dev might do and restarted the server. Bad mistake! The MySQL server refused to come back up. Just totally refused. After some time …

%d bloggers like this:
Skip to toolbar