[Previous entry: "Those Wacky Supreme Court Justices..."] [Next entry: "National Film Challenge"]

There are two type of people reading this post. One type, the group which includes my wife (not that she reads this blog, but for example's sake), will care so little about this subject that they might actually yawn and fall asleep on the spot. When I mention RAID, they will think of bug spray. The other, a group which will include my computer geek friends, might need to grab a bib. The slobber potential is extremely high.

I'm building a home RAID. And to kick it off, I just bought a Terabyte of hard drive space.

To make a long story long, I've been having trouble with my main video editing computer for some time now. It's been flakey, shutting down randomly sometimes, video captures failing for no apparent reason, and iffy audio recording. This has caused setbacks in various projects, including wowgirl and some other smaller things. This came to a head a few weeks back when I started working on the wedding video that I shot for my friend Mike's brother. I was having real problems pulling the tapes and it took several tries (at an hour a try, mind you) to get each tape pulled without the machine locking up.

On top of this, I keep running out of space. I'll go out, buy another 80GB or 160GB drive every now and then, and it doesn't matter, I fill it up so fast. Each wedding runs 50-70GB easy. And the question of backup? wowgirl is about 60% backed up on a huge stack of DVDs, but I wanted something more reliable and less pain in the ass to deal with (ripping those DVDs is time consuming).

Finally with all the problems with this wedding video, I decided to deal with the situation. I determined that I mainly have three problems.

1) Too many hard drives, not enough space. I have something like seven hard drives, 5 internal, 2 external. This is causing heat problems and data dislocation - wowgirl is literally spread across 6 different drive letters. And I need to add more, I'm almost completely out of room.

2) No decent backup capability. I keep everything I do. I still have both Duck and Reuben's wedding data on one of my drives, as well as pretty much every piece of video for everything I've ever worked on (music videos, 48 Hour Film project data). And the something like 350GB of wowgirl data. I've got a good part of it backed up, but it's time consuming to deal the DVDs and the lengths of my files involve spanning, and the longevity of a burned DVD is a big question mark. And if any one of these drives decided to die, I'd have a bit of a problem on my hand.

3) The crashing issue. The random crashing issue is the most immediate problem I have - it's keeping me from working on things. My best guess was that the crashing was being caused by the heat being produced by the 5 drives mounted in my server case along with my Dual Athlon board. A couple of other possibilities could also be to blame.

Temporarily, I've put some band-aids on the system. I dealt with the crashing issue by removing some of the hard drives from the current system. Of course, I don't have access to that data for the moment, but the removal of the heat has made the system much more stable The heat was probably the cause of the crashing, as I have yet to experience a crash since I did that (knock on cyberwood). I also bought a more expensive, certified power supply to make sure the motherboard is getting a good, steady power stream. Lack of temp space might have been an issue with Premiere (the capturing and crashing during editing) as well, though. I fixed that (briefly) by picking up a cheap hard drive at Fry's (120GB, ended up being $20 after rebates - a steal).

But I wanted a solution that wasn't temporary. I talked to my wife and we decided on a budget ($400 initially - it ended up closer to $550). I researched tape backups (and I still may add one in the future, but for now, they're not right) and other backup mechanisms. Too pricey - sometimes the tapes cost $50 each.

So I started looking at cheap hardware RAID 5 setups. (For the uninitiated, here's the wiki on RAID in general, you can click on the link there for RAID 5 specific stuff.) The most important aspect of RAID 5 is the fact that a drive can die, and you can replace the dead drive with a new one, with no loss of data. There's a speed boost too. 3ware makes a good hardware RAID card, as does MSILogic and a couple of other companies. But in order to get enough ports to ensure future expandability (8 at least), I'd have to buy a $350-400 card up front. Can't very well kill the budget without buying the drive, now can we?

But in my research, I discover that Linux has some powerful software RAID tools. The benefit of Linux based software RAID is that I can add additional drives at any point (as long as I have spare ports, and the four port SATA card I bought ran about $60). Plus, if the hardware controller dies, the RAID setup still exists. Just replace the ports, and the RAID is back up. Nice.

I bought four Western Digital SATA 250GB Hard Drives to kick it off. A damn terabyte.

(The astute among the computer geek side are now realizing that I really only have 750GB of space, not a terabyte, and are laughing at me. To those people I say, please kindly go service youself with 36" crowbar. I can always add more drives to the array).

Here's how I'm doing this at home, for such a low budget. I have a PC here that I have Linux installed on (though I'm going to reinstall to one of the RedHat Enterprise clones). I was originally goin g to use it for Ardour, a digital audio workstation, but the sound card problems and the fact that all of my friends in other locations are using Cakewalk software means that I'm going to be using Cakewalk on the big editing box and not worrying about the Linux solution. For now, anyway.

So I'm taking this Linux PC I already own and turning it into a bad ass File Server. It's a 1.3Ghz Pentium 4 (the only non-Athlon machine I own, and there's a long, drawn out story of why I even have a Pentium, since I tend to despise Intel). I'm going to use mdadm 2.0 to manage a linux software RAID. I plan on using XFS for a file system, mostly based on the fact the the DISH Network Hi-Def PVR uses it to handle massive amounts of Hi-Def on a single drive. JFS would also be a workable solution.

Here's the entire hardware arrangement, including links to the items I bought, in case you are interested in any of the parts:

1.3Ghz Pentium 4 on an Abit TH7 motherboard.
768MB of RDRAM
80GB IDE boot drive

(1) Promise SATAII TX4 4-port adapter - $58 at newegg.com.

(4) Western Digital WD2500JD 250GB SATA hard drives - $93 each at bestbargainpc.com. New drive, 3 year warranty, 250GB SATA and a grand total of $372. Free shipping, no less.

To make sure I had enough bandwidth between machines, I took advantage of a CompUSA sale and picked up a Netgear 5-port gigabit ethernet switch for $20 after rebates. I grabbed a couple of Netgear gigabit ethernet cards for $15 each from JR.com and I'll pop one in the editing box and one in the file server. I also picked up a couple of Cat6 patch cables, just to help ensure strong data transfer.

Most of these parts are still in the mail. Once they arrive and I get them set up, I'll let you know how it works. Or I'll be begging for help if I can't get the RAID setup right.

Replies: 8 Comments

on Monday, September 19th, Vampire Elf said

Yeah, I gave you a copy of everything, the edited ceremony plus all the raw, unedited footage. Of course, that was back before DVD burners, so it was on VHS, but since I still got it all, I can crank you out a DVD of the wedding if you want. Since the source was at least half miniDV, it'll probably look a little better than the VHS copy.

RAID 0+1 is badass. But it doubles the drive cost. Can't do it right now.

I haven't really bought a new machine since I bought the Dual Athlon board, which was several years ago. Added some RAM and the second processor last year. The hard drives are the current bottleneck. And I think the new job has a lot to do with my being able to purchase the hardware.

Email me your phone number, too. I've changed phones since last we talked and your number is MIA.

on Saturday, September 17th, Kicking Bird said

Speaking of the Duck wedding...did I ever get a copy of that?

At least your wife allows you to spend money on new hardware. The fastest PC in my house is a Gateway server that has 800 Mhz Pentium III procs in it, given to me by a colleague who felt bad for me and my previously-schmoking! HP Celery-800 PC.

Dude. Go RAID 0+1...That is baaaaad-ass!

on Thursday, September 8th, Vampire Elf said

Right on, Scruggy! I will be there. Where is there, by the way?

Ape Man - Yeah, I may have to eventually, though for the moment, I'm going to continue to back up important stuff on DVDs, even though it sucks. Couldn't do the array and the tape drives together, I'm already busting the budget. DDS-4 is the only affordable way to go, thanks to Ebay, and you can maybe get a little more than 20GB with compression. My biggest files are running 13GBs each, but the problem is each project has an average of 4-5 of those files. So I either have to split the files across multiple tapes or use one tape per file - neither is a great answer. I was kinda thinking I would have out and try and get a DDS-5 (it's got some new name, like DT72 or something), the drives are a little more, but you jump to 36GB uncompressed, and hopefully the tapes won't be too bad. We'll see how it goes.

For now, I'm going to hope the array works out.

on Thursday, September 8th, beaumontkitty said

I cannot believe i just read every freaking word of this post- including the comments! I've never felt dumber in my entire life!

And you make BOXES? Why don't you work for NASA or something...geeze!

on Wednesday, September 7th, Scruggy said

Sounds like you know a thing or two about computers. Could you fix mine so it doesn't crash everytime I complete a quest in World of Warcraft? Thanks. Hey, September 23rd is a Friday night! Come on out to the big show.
--Bert Bert

on Wednesday, September 7th, The Ape Man said

BTW never mind on the TR-5's. Those little 10/20GB turds are still selling for $40 each. What the hell? TR-5 is shitty anyway.

on Wednesday, September 7th, The Ape Man said

OK, I lied.

A hundred bucks.

And a box of ten tapes for $59.

on Wednesday, September 7th, The Ape Man said

RAID 5 is indeed nice. However you still want to keep some kind of backup of anything you really can't live without. Here's why:

1) You really can lose two drives at once. This actually happened to me on a server that I had wrestled with for a year to get working properly, and one day I came in and there are all these lights flashing on the RAID 5 disks. I booted to the RAID controller and saw that indeed, two of my drives had failed on the same day, rendering my parity bit (and my array) useless. Really unlikely, but not as unlikely as you'd think if you don't think of the fact that the five drives in the array came off the same assembly line probably within minutes of each other, so the odds against two of them failing at roughly the same time are actually not that astronomical.

I had a backup, but of course it never really worked properly again after I restored everything, and at that point I didn't have the energy. Fortunately the project that used that piece of junk got canceled, so now it's basically a vanity machine for one of our most useless project managers (he likes to use Oracle for everything because he wants to be an Oracle guy when he grows up.)

But that's neither here nor there.

2) In theory, if your controller goes, and you have software RAID, you're cool once you get a new controller. But beware - when a controller goes, depending on how it goes and why, it can take drives with it. If that happens you'll wish you had a backup.

The good news is, if you are creative you should be able to get a pretty cost-effective backup solution. For example, you could probably buy an old DDS-4 or TR-5 drive off Ebay for fifty bucks, and those tapes are really cheap. They only hold like 20 gigs, but again, this is for the system config and any data you would die if you lost, not everything.