I keep noticing startups (and larger companies, too) making some really basic mistakes that end up leading to a lot of downtime. Here's my list of 6 really easy things you can do to avoid major downtime.

1. Buy backup DNS service
This is so cheap it's a no-brainer. For about $15/year you can get a service that will constantly grab your DNS data and act as a backup if they happen to go down. Otherwise, when your DNS servers go haywire (it's happened to me and I've seen it happen to many others), you'll be stuck helpless for a few hours as people are unable to get to your site. [I've used No-IP Squared Backup, and Chris has used Nettica].

2. Buy a monitoring service
For $5/month, you can purchase a service that pings your servers every few minutes and sends you a text message if they go down. This is absolutely crucial, especially if things go to hell in the middle of the night, or any other time you might not normally be checking. Make sure to buy a service that monitors from at least 3 locations -- there's nothing worse than a few false alarms in the middle of the night, after which you won't get up for the real thing. [I've been happy with WebSitePulse -- their prices are a bit more expensive now, but you should be able to get them down to $5/month on the phone.]

3. Always make database backups before touching the database
It's one of those things you always consider and dismiss right before you bring the whole thing crashing down. Especially if you aren't making very regular backups (note: you should be), make sure to do so before you get your hands dirty. (Ever forget the WHERE clause? Not fun...)

4. Be VERY careful around power cords at Colos
Knocking out a power cord seems to happen consistently if you don't make a very concerted effort not to. It's extremely easy for a cord to jiggle a tiny bit, or for one moving server to pull on another cord in just the right way. Always plan out your server trajectories before you move them, or have someone to hold the power cords in. [Note: Why aren't snap-in power cords standard for rack mountable servers??]

5. Make your site functional in pieces
Even if your database is down, there's no reason your home page shouldn't still show, or any of your other static pages. There's a big difference from a user's point of view in between an otherwise seemingly functional site that shows a nice looking error message, and a site that spits out errors, is not accessible, or won't load at all. If Weebly's database goes down, users will see a polite "Sorry, something is wrong and we're fixing it right now" error message. Our site, blog, and all hosted user sites stay up, so a database crash just means that people can't edit their sites at the moment.

6. Use source management to roll out updates
We use darcs to manage our different source repositories, and it's a flexible and distributed system that works very well for us. Whatever you use, make sure you use some automated process to roll out updates (which doesn't include moving a directory and moving another in it's place, and, God forbid, manually diff'ing files -- there's always more to that than you anticipate). It's quite shameful to see pretty basic sites go down for hours (or days, or weeks) rolling out an update. If you're using Weebly while we push out an update, your session will automatically be refreshed without any loss of data, and you'll be up and running on the new version within seconds (with no downtime). [Darcs can be found at http://darcs.net/]

Those 6 items combined have probably caused over 80% of downtime I've been responsible for. What's your list?

14 Comments
 
Work vs. Value 08/21/2007
 

One of my biggest frustrations with academia was the tendency to place emphasis on work. I've heard that this can be different at other institutions, but most people I've talked to generally agree that emphasis in college was placed on work.

Here's a recurring example: I'd often get a lower grade than other peers who "worked harder", even though my final grades or output were very clearly of a much higher quality. I generally didn't have much use for going to class, as I could learn what was presented much more efficiently on my own.

Not that I really cared about getting a B+ instead of an A -- if I did, I would have gone to every class. But it seems like the emphasis on work gives students the wrong priorities.

There seems to be two inputs to value: work and ability. If you have less ability, you can compensate by working harder than average. And if you have above average ability, you may tend to work less.

Note that ability doesn't necessarily directly translate to intelligence, and that I'm not downplaying hard work: those who both work hard and have ability will produce the most value.

But by putting emphasis squarely on work, academia is punishing those with above-average ability. My experience in college was that both the top 5% and bottom 5% of any given class did the least work, but doing less work was uniformly regarded as bad.

In the real world, though, value is most important. As an example, let's take two people who make pottery. One is a natural artist, and makes beautiful pottery. The other tries really, really hard, but the pottery isn't great. It may not be "fair" to the person who tried hard, but the beautiful pottery will be sold for much more money, as it's of higher value. What really matters to people is how much value you are providing them, not how much work you put into it.

I'm also not saying that ability can't be learned: in the above example, ability may represent both natural artistic abilities and learned skills.

It seems like a much better system would be to judge on value, and compare the final output. Those with less ability would be required to work harder to produce the same value -- they won't be taught that hard work without value is OK. And those with above average ability wouldn't be weighted down performing bullshit work; they'd have more time instead to focus on projects more interesting and useful to them.

14 Comments