Sunday, August 07, 2005

Step-by-Step to Windows RAID using CoLinux

Windows XP Pro does not support RAID-5 through software without some system level hacks, and even then will not allow you to build an array on external USB-2 drives. Windows XP Home doesn't even support those options.

So, if you had a large enough box, you "could" get a RAID card, although RAID 5 cards aren't particularly cheap. However, you would be doing well to get more than a couple drives in a box that wasn't intended to be a server, and you'd have to worry about the additional heat and power drain.

For those reasons and others, you might prefer to setup RAID using a number of external disks. And the best way I've found to do that is to use coLinux to host a Linux Software array and serve it up with Samba.

REQUIREMENTS

To begin, you will need to have downloaded the following:
coLinux 0.63rc-2

coLinux 0.7.1 snapshot (Get all 5 files shown here)

Debian image (Debian-3.0r2.ext3-mit-backports.1gb.bz2) from the 26images page at the coLinux wiki


Swap image from the AddSwapPartition page at the coLinux wiki

You'll need to begin by installing the coLinux 0.63.rc-2 system to your computer since the 0.7.1 snapshot doesn't have a complete distribution or an installer. You need to install the base system and the TAP driver, but you don't need to install the Winpcap files, nor do you need to DL the Debian image again.

Once you've installed 0.63-rc2, you should unpack the 0.7.1 snapshot, and copy the files over from the directory you unpacked in to the directory you installed 0.63-rc2 in. You would then want to unpack your debian image in this same directory. Either WinZip or 7-Zip can unpack the file. Be aware that while the download is only 30 megs, the unpacked version will take up over 1 gig of space. Make sure you have room on your drive.

NETWORKING

While slirp is the easiest networking protocol to get working with coLinux (it pretty much "just works"), it is only good for letting coLinux talk to the outside world. Since we will need to "serve" our array, we have to use a two-way connection. In my experience, that works best with the TAP driver from 0.7.1.

At this point, you would want to Open Network Connections either by right-clicking on the icon for your current network connection, or by going to Control Panel->Network and Internet Connections->Network Connections. Once there, you should see at least one Local Area Connection. If you have more than one, putting your mouse over each one will identify which device is the TAP network connection. The TAP network connection should be set to an IP address that isn't currently being used by your network - for myself I use 192.168.0.155.

You will then want to set the net mask of 255.255.255.0. Do NOT set any of the other options for the following to work - I recommend using the CTRL key to select the TAP connection, and whatever connection you normally use to connect to the internet, and then choosing to Bridge Connections. I would point out that Windows will often then show the other network connection as being offline, even though it usually isn't. A reboot or right-clicking and choosing "repair" will often set things straight.

COMMAND LINE CONFIGURATION

Since you are using 0.7.1, you won't be using the XML configuration files, but instead will be using either command-line options, or a command-line like configuration file. I was getting Windows errors using a configuration file, but your mileage may vary (YMMV). Since you don't want to have to type everything over again and again, I recommend creating a batch file with all of your options in it. A typical initial setup looks like:

colinux-daemon.exe mem=128 kernel=vmlinux cobd0=c:\coMake\Debian-3.0r2.ext3-mit-backports.1gb cobd1=c:\coMake\swap_512Mb root=/dev/cobd0 eth0=tuntap,"Local Area Connection 3"

Note that this should all be on one line even though it may appear wrapped here. You'll notice that I have installed coLinux to a directory called "coMake". You'll be making changes as we go along to reflect the additional files you will need for your RAID array, but this is good enough to start.

STARTING UP

Save your BAT file, and you're now ready to run coLinux for the first time. Double-click the BAT file, and a command line window should open, followed shortly by a terminal window called "Cooperative Linux console". Assuming you don't have any corrupt or missing files, you should eventually get to a screen with a login prompt. Type in:
root
and hit the enter key. This will prompt you for a password, where you should type in:
root
and hit the enter key again. I highly recommend that once you get your system stable you should change your password. At this point it's not quite such a good idea since you will be changing a lot of other things around, and you'd hate to start over because you get confused as to what your password is.

At this point, your main task is to get your network connection working. To do that, you'll want to enter:
editor /etc/network/interfaces
and press the enter key to bring you into the editor with the file open. You'll want to edit it to look something like this:

# The loopback interface
auto lo
iface lo inet loopback

# The first network card - this entry was created during the Debian installation

auto eth0
iface eth0 inet static
address 192.168.0.153
netmask 255.255.255.0
gateway 192.168.0.1

It doesn't hurt to leave any of the other comment lines in the file. The main things you will change are the address and the gateway. The address should NOT be the same as any address on your LAN. It's an easy mistake to think this should be the same address as your TAP adapter. But that's not how it works.

For the gateway, you will want to use the gateway your system is using to get to the internet. If you're not sure what it is for your system, open a Windows Command prompt and type in:
ipconfig /all
and then the enter key. A number of lines will scroll past - one of which should be "Default Gateway". The address there is the one you want to use here. If you're using the default editor, then use Ctrl-X to exit the file. You will be asked if you want to save your changes, so answer Y for yes. Editor will then tell you it's saving the file as the original file name, and then you should be taken back to the coLinux command prompt. Now it's time to edit /etc/resolv.conf to setup your DNS.

When you ran ipconfig /all, you should have gotten one or more addresses next to the line DNS servers. For each of these, you would want to add a line, similar to my file below:

nameserver 24.93.41.125
nameserver 24.28.131.62
nameserver 24.28.131.63

Save your work, and when you get back to the coLinux command prompt, type in:
ifdown eth0
and enter, and then when the next coLinux command prompt comes up type in:
ifup eth0
and enter. If everything is setup right, you should be able to now type in:
ping google.com
and enter, and see that packets are getting back and forth to your coLinux instance. Hit "CTRL-C" (that's holding the Control and C keys down together quickly) to stop ping from testing the connection. It will eventually stop on it's own, but there's still more work ahead.

EXPANDING YOUR SPACE

The first step is to give your coLinux instance some more memory space by getting your SWAP file up and running. The first step is to type in:
mkswap /dev/cobd1
and press enter. Once this completes you want to type in:
swapon -a /dev/codb1
and press enter, followed by:
sync; sync; sync
and press enter. If you now type:
free
and press enter, you should see your swap partition listed. If everything went right, then you will want to edit your /etc/fstab file to make sure that the swap will be loaded at boot time in the future. Add the following line to the file:

/dev/cobd1 none swap sw defaults 0 0

and then save it. Note that you actually type the word "none" in rather than just leave it blank. A mistake I made myself initially :-)

While a one gig root partition is ok, you would like to make sure that you aren't going to run out of space once you start adding packages and files. If you are just going to follow the instructions to get RAID up, this step isn't absolutely necessary. But if you are going to setup an X Server or a KDE desktop, then space will be at a premium. For me, I followed the steps given at the wiki page "ExpandingRoot" following the "mksparse" instructions at the bottom of the page. The other examples given might work, but that's not what I did, so I can't say for sure.

You will end up shutting down your coLinux instance (using "shutdown -h now"), and then bringing it back up as part of this process. Make sure that you are editing your BAT file and not a config file to use the new root partition, or you won't see your changes.

Your BAT file should now look a bit more like this:

colinux-daemon.exe mem=128 kernel=vmlinux cobd0=c:\coMake\5gigfs cobd1=c:\coMake\swap_512Mb root=/dev/cobd0 eth0=tuntap,"Local Area Connection 3"

Again, that should all be on one line even though it is wrapped here.

ADDING PACKAGES

At this point you already have a decent coLinux system, but you are a good ways from being able to setup your RAID system. I mention this because some folks seem to feel they have to do everything all at once. However, you might find it fun to spend some time just learning how to use Linux commands like "ls -al" to get a directory listing or "top" to see all the processes that coLinux is already running.

Having taken a breather, it's time to start getting the various packages you will need in order to setup a custom kernel, and to eventually get your RAID array created. The first step in doing that is to edit the /etc/apt/sources.list file. You want to make it look similar to the following:

deb http://ftp.us.debian.org/debian/ stable main contrib
deb-src http://ftp.us.debian.org/debian/ stable main
deb http://security.debian.org/ stable/updates main

You will probably be deleting at least one line (the backports one) that is in the default version of this file. The sources that you are going to get your packages from are important because you want fairly recent versions of tools, but not the MOST recent versions. Not only have these versions been tested a bit more, but coLinux 0.7.1 checks to make sure that your kernel was built using gcc-3.3.x (a compiler you'll be installing in a minute), and the most recent version of gcc is 4.0.x. Given the time this takes to setup, you really don't want to have to go through it all again just to recompile with the right version of gcc.

After you've saved your edited file, you now want to type:
apt-get update
and press enter. This will build a list of different packages that are available at these sites. Once that has completed, type:
apt-get install wget
and press enter. This adds a tool that is used for getting other tools. You may be asked if it's ok to install or update some files. If so, answer yes.

Next, type:
apt-get install build-essential
and press enter. This is going to add a number of tools you will need for building your custom kernel, or for programming, or even to rebuild packages from source if you need them. You may be asked if it's ok to install or update some files. If so, answer yes.

Now, type:
apt-get install gettext bison flex automake autoconf make patch
and press enter. This installs some more needed tools. You may be asked if it's ok to install or update some files. If so, answer yes.

This time type:
apt-get install unzip bzip2
and press enter. This adds some needed tools to work with archives. As always, if you're asked if it's ok to install or update, answer yes.

Now, you need to type:
apt-get install ncurses-base
and press enter to add a tool used for drawing to the screen.

Then type:
apt-get install ncurses-bin
and press enter to add the tools used by ncurses.

And then type:
apt-get install ncurses-dev
and press enter to add the headers and source code that are used.

You're almost done, but will need to type:
apt-get install samba
and press enter to add the service that will make your coLinux files (and the RAID array(s) available to Windows). As it is installing, it will ask you some questions about how to configure samba. The first question will be what WORKGROUP or DOMAIN your LAN uses. I recommend using a Workgroup, and using a name different than the default one that Windows sets up (WORKGROUP). Be aware that the name you use here will need to be the same one your Windows session is using for your computer. That may mean you have to change the name used in Windows if you let it use WORKGROUP. If you're not sure what your Workgroup name (or Domain name if you're using that), right-click on My Computer (in Windows) and select "Properties". Then click the "Computer Name" tab to see what you are currently using. This is also where you can change the name if you want. However, if you change the name, Windows will want to reboot your machine, so don't change the name before you complete this section. And even then, make sure you do a controlled shutdown of your coLinux instance before letting Windows reboot. For all the other samba configuration options, always accept the default answer.

The last step, in this section, is to type:
apt-get install smbfs
and press enter to add some filesystem tools needed by samba.

You have probably taken a bit of time to do all this, so you might want to take another breather before starting the next section. This is also a good point to shutdown the coLinux session and reboot Windows if you need to change your workgroup name.

CONFIGURING SAMBA

The main reason we are using coLinux 0.7.1 rather than 0.6.3rc-2 is that the combination of the reworked TAP driver and the way coLinux works with Samba under 0.7.1 is much, much faster. Under 0.6.3rc-2, a 780 meg file takes 8-9 minutes to copy to a coLinux samba partition. That same file takes 2-3 minutes with 0.7.1, and is almost as fast as the speed it takes to copy the file from one disk to another under Windows natively (about 2 minutes).

However, there are some things we need to do to make samba work at all, and some things we need to do to make it as fast as we can without affecting data safety. To begin with, we need to create a directory that we're going to share to move some files back and forth. Type the following three lines, pressing enter after each one:

mkdir /usr/src
chmod 777 /usr
chmod 777 /usr/src

It is important to note that you are turning off the security to a good bit of your coLinux file system by doing this. Since the coLinux system is actually running within a Windows session, this is not as dangerous as it might be. However you want to make sure that you either have a firewall setup on your router, or that you are using a software firewall in Windows if you are going to be doing this. You don't HAVE to do that, but it reduces the already slim chance of having problems to almost 0.

Next, use your editor to open the /etc/samba/smb.conf file. You will want to scroll down to a line that looks like:

; security = user

and change it to:

security = share

This tells samba that you don't need to log in to access the files on the directory you are sharing. This is a bit of a security risk, which is part of the reason I suggest not using the default Microsoft Workgroup name of WORKGROUP. However, this not only makes accessing the files easier, but buys a very slight performance upgrade since sessions aren't trying to maintain security.

A couple lines further down is a line like:

invalid users = root

which you change by adding a semi-colon in front of it to be:

; invalid users = root

Finally, at the end of the file, you will want to add the following lines:

[usr_src]
path = /usr/src
public = yes
writable = yes
browsable = yes
create mode = 0777
directory mode = 0777
guest ok = yes

You can then save the configuration file. If everything has gone right, you should be able to go to My Network Places->View Workgroup Computers and see a coLinux computer. Double-click on it and you should be taken to "usr_src" which is your /usr/src directory in coLinux.

BUILDING A KERNEL WITH RAID

You are now ready to start building the custom kernel you will need to make the RAID tools work. To do this, go to your Windows session, and navigate to:
http://www.kernel.org/pub/linux/kernel/v2.6/

Scroll down that page, and download the file:
linux-2.6.11.9.tar.bz2

While some of the other kernel source files might work, this is the one I used, and so it's the only one I can say with some confidence will work. Once it is downloaded, you will want to copy it from where you downloaded it to "usr_src".

At this point you should also go to where you unpacked your coLinux 0.7.1 snapshot. There should be a directory there called:
colinux-20050720

Go into that directory, and look for the conf sub-directory. Copy the file called linux-conf to "usr_src". Now go to the patch sub-directory within colinux-2005-07-20 and copy the file called linux-2.6.11.diff to "usr_src".

Go back to your coLinux session, and type in:
cd /usr/src
and press enter to navigate to the /usr/src directory. You should see all three of the files you just copied there.

Now, type in:
tar -jxvf linux-2.6.11.9.tar.bz2
and press enter. This will unpack the linux kernel source code, and usually takes a few minutes.

When it is done, you will need to type in:
ln -s /usr/src/linux-2.6.11.9 linux
and press enter. This sets up a "symbolic link" between the generic /usr/src/linux directory that many tools expect the kernel source to be in, and the actual directory where the source code is. This setup makes it easy to work with multiple revisions of a kernel by just changing where the symbolic link is pointing.

Next type in:
cd /usr/src/linux
and press enter to make sure you are in the correct directory.

Now, type in:
patch -p1 < ../linux-2.6.11.diff
and press enter. This makes your kernel source compatible with what the coLinux daemon expects when it runs your compiled kernel. It saves you from editing a lot of files by hand.

For the next step, type in:
make mrproper
and press enter. This cleans up the environment from any extra files that might be in the directory from trying to do a build earlier, and prepares a default config file which you're about to replace.

To do that, you will type in:
mv linux-config .config
and press enter. This renames your linux-config file so that it will be used by the kernel build process.

Now, type in:
make menuconfig
and press enter. This takes you into a graphical interface for specifying how you want your kernel to work. This is generally NOT a place to start changing things that you aren't sure about. Since we're only trying to add RAID support, arrow down to Device Drivers, and press enter.

You now are in another menu where one of the options is Multi-device support (RAID and LVM). Arrow down to this, and press enter. This now gives you a box where you will want to press the space key to make an asterisk show up, and which will then cause a number of other options to show up. You want the asterisk to be there so that the support is built in, rather than requiring you to load the module at boot - and then having to do some additional work.

From here, you can select which types of RAID support you want built in. Since I usually use RAID-5, but like to have the option to do some other things, I selected RAID-0, RAID-1, and RAID-5. For all of the options you choose, make sure that you have an asterisk and not an M in the place where you indicate your preference.

Once you are done here, hit the escape key twice to exit this menu. Hit the escape key twice again to exit the configuration process. You will then be asked if you want to save your changes. Make sure to select YES here.

The next step takes anywhere from 20 minutes to several hours depending on how fast your computer is. Since you can't shut down coLinux while this is going on, make sure that you aren't doing anything that would interrupt the process or require a reboot of coLinux or the system.

If you're ready, then type:
make vmlinux
and press enter. This begins the process of creating your new kernel. As mentioned above, this is a lengthy process. Don't be surprised if it keeps going, and going, and going...

Usually it's pretty obvious if it's actually stuck on something, and the compile is designed to abort if something's important, and to ignore if it isn't. So if you see some warnings, don't get too upset as long as the build continues.

When you get back to your coLinux command prompt, you have one more compile step. So type in:
make modules modules_install
and press enter. This is another lengthy compile, although it usually runs somewhat faster than the previous one.

When the coLinux command prompt returns, you now have a custom kernel ready to be used. But remember, you're already running coLinux, so you can't just copy it directly to your coLinux home directory. Instead, go back into Windows and bring up "usr_src" and copy the file "vmlinux" anywhere BUT your coLinux home directory. The Desktop isn't a bad place for this since you're going to move it again anyway.

You now need to shutdown (shutdown -h now) your coLinux instance. Once it is shut down, rename the vmlinux in your coLinux home directory to vmlinux.old, and then copy your new vmlinux file to the coLinux home directory.

Double-click your BAT file, and if everything is setup correctly, you should now be running coLinux with RAID support built in.

PREPARING FOR RAID

You're getting pretty close to having your RAID array (or arrays) up, but there are still a few steps you need to follow. Be aware that at this point the actual type of RAID and your personal setup will require you to read about what I've done, and then think about how it would apply to your setup. IOW, while some steps should be done exactly as I show them, a lot of them need to be modified to suit your setup.

The first step is easy enough. Type in:
apt-get install mdadm
and press enter to install the RAID management tool of choice. The mdadm tool has pretty much replaced the older raidtools toolkit, and certainly is my choice of tools.

However, the next step from here depends on what type of RAID you're creating and using which disks.

To begin with, you are likely going to have several disks, and therefore will need more than the default 4 cobd devices that come with the Debian image. In my case, I had 2 200 Gig Hard Drives and 4 250 Gig Hard Drives. Because I had quite a bit of data already, I couldn't just use all of the space on all of the drives as one array. Besides, arrays need to be build of arrays of the same size. Also, it is less "flaky" to use mksparse files than to try and map the drives directly. So, I ended up going with 3 arrays. The first array was built with the 50 gigs "extra" on each of the 250 gig hard drives. Specifically I created:
50gigfs_j
50gigfs_l
50gigfs_m
50gigfs_n

Each file was created by using the windows command prompt to go to the hard drive I wanted the file on, and then entering the full path to the mksparse executable and the file name and size. So, for example:
c:\> j:
j:\> c:\coMake\mksparse\mksparse 50gigfs_j 50G

and then repeat as necessary. I didn't have to name the files with those specific names, but it did make it easier for me to know which file on which drive I was working with. That left me with six drives each of which had 200 gigs of free space on them. Since I was going to have to move files around, I knew that I would be creating two 100 gig files on each disk to make two arrays - one from each set of 100 gig files. This meant that I would have 16 files altogether that I would need cobd devices for. The easiest way to create the cobd files is to type something like the following at the coLinux command prompt:

for i in 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19; do mknod /dev/cobd$i b 117 $i; done

and then press enter. If you then type:
ls -al /dev/cobd*
and press enter you should see as many cobd files as you created, plus the four that were already there (which is why I started at 5).

You will also need a mount point (coLinux directory) for your array. I personally would recommend something like /mnt/md0 which you would create by typing:

mkdir /mnt/md0

and pressing enter. Note that you will need to then set the permissions to allow your array to be shared with Windows using samba. To do that you need to enter the following two lines, pressing enter after each:

chmod 777 /mnt
chmod 777 /md0

And you will also need to edit /etc/samba/smb.conf to add a section similar to the following at the end of the file:

[md0]
path = /mnt/md0
public = yes
writable = yes
browsable = yes
create mode = 0777
directory mode = 0777
guest ok = yes

Once you've done all this, it's time to shutdown coLinux again. Once it's shutdown (using shutdown -h now), you will need to edit your BAT file to associate the new cobd files with the new mksparse files. It is important not to map files that haven't been created yet. So, given the setup above (which covered my first array), you would end up with something like:

colinux-daemon.exe mem=128 kernel=vmlinux cobd0=c:\coMake\5gigfs cobd1=c:\coMake\swap_512Mb cobd3=j:\50gigfs_j cobd4=l:\50gigfs_l cobd5=m:\50gigfs_m cobd6=n:\50gigfs_n root=/dev/cobd0 eth0=tuntap,"Local Area Connection 3"

BUILDING YOUR ARRAY

Double-click your changed BAT file, and you are ready to actually build your array.

Continuing the example above, you would enter an mdadm command similar to this:

mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/cobd3 /dev/cobd4 /dev/cobd5 /dev/cobd6

and press enter. This should all be on one line, even though it might wrap on your display. Most of these options should be fairly obvious. However, you can learn more about mdadm at:
http://www.linuxdevcenter.com/pub/a/linux/2002/12/05/RAID.html


Your system will spend some time building the array, but you don't have to wait for it to finish to start using it. For example, after it has gotten started, you can do the following to format the array:

mke2fs -j /dev/md0

and press enter. This creates an ext3 filesystem on your array. Since you want your array to be automatically brought up in the future whenever you start up your coLinux session, you need to edit your /etc/fstab to add a line like:

/dev/md0 /mnt/md0 ext3 defaults 0 0

You also need to create an mdadm.conf file that the system will use to know which devices go to which array. The easiest way to do this is to type in the following line:

mdadm --detail --scan > /etc/mdadm/mdadm.conf

This creates a file that is almost like what you want. The file will initially look like:

ARRAY /dev/md0 level=raid5 num-devices=4 UUID=777b15e6:6791bb61:96a9c2f4:80e2f7bf
devices=/dev/cobd3,/dev/cobd4,/dev/cobd5,/dev/cobd6

You need to edit it to look like:

ARRAY /dev/md0 level=raid5 num-devices=4 UUID=777b15e6:6791bb61:96a9c2f4:80e2f7bf
DEVICE /dev/cobd3 /dev/cobd4 /dev/cobd5 /dev/cobd6

At this point you can mount the array by typing:

mount /dev/md0 /mnt/md0

and pressing enter. You will eventually get a message on the coLinux console telling you that the md0 sync has completed. You don't have to wait for that before you can start copying files to it, but your read performance from it will be terrible until it's done. And again, you don't want to shut down your coLinux session until the sync has been completed.

FINAL THOUGHTS

In my case, I had to go through several steps to get all my arrays built, and building my kernel took several starts and restarts. This guide is designed to help you avoid at least some of the problems I encountered. Be aware that there may be specifics of my environment that I didn't catch in writing this, and so you may want to think through what is being done rather than just follow this step by step blindly.

Of course, if you've found this guide helpful, or if you have questions, I'd be more than happy to get an email at:

ewan dot grantham at gmail dot com (mild obfuscation to throw off the spam folks).

Post a Comment