Updated! 6/6/2008

I posted the amd64 version of the compiled DRBD 8.2.5 driver at the end of this post!

DRBD is a block device driver for Linux that allows you to mirror a partition between two servers.

I had a single application server, but whenever a server failure occurred, my websites, Subversion repository, and e-mail would go down. In order to be highly available, I added a second server in the event of a failure.

app1.cb1inc.com app2.cb1inc.com
AMD Opteron 180 2.4GHz dual-core
4GB RAM DDR
2 x 250GB SATA 7,200 hard drives (Software RAID 1)
2 x Gigabit network cards
AMD Sempron 2800+
3GB RAM DDR
250GB IDE 7,200 hard drive
2 x Gigabit network cards

I don’t have a ton of load, so the second server doesn’t have to be super powerful.

Each machine has 2 network cards: one for public and one for private traffic. The public interfaces connect to my main switch on the 192.168.0.X network. The private interfaces are connected via a crossover cable on the 10.26.210.X network.

Once you have installed Ubuntu 8.04 Server, you need to install some build tools. I don’t know specifically which build packages you need, so I just install a bunch of them and it should work. 🙂

apt-get install build-essential binutils cpp gcc autoconf automake1.9 libtool autotools-dev g++ make flex

In theory, that should get all of the development tools installed that you’ll need.

Next we need to get the entire kernel source code. Just the kernel headers won’t cut it. We need to compile the kernel so that it builds some of the scripts needed to compile the DRBD driver. We’ll also install the ncurses library so that menuconfig works.

apt-get install libncurses5-dev linux-source-2.6.24

Then extract the kernel source:

cd /usr/src
tar -xvf linux-source-2.6.24.tar.bz2
cd /usr/src/linux-source-2.6.24

Next, lets clean up any unneeded files (which there shouldn’t be any the first time):

make mrproper

Before you can build the kernel, you need to copy your existing kernel build configuration into the kernel source directory:

cp /boot/config-2.6.24-16-server /usr/src/linux-source-2.6.24/.config

Now we run the menuconfig which will read in our kernel build configuration and build some version files. As soon as the GUI appears, just exit immediately. You don’t have to change any of the settings.

make menuconfig

Finally we need to prepare the kernel and compile it. This will take quite some time.

make prepare
make

Now that we have the kernel source compiled and ready to go, let’s get the DRBD source.

cd /root
wget http://oss.linbit.com/drbd/8.2/drbd-8.2.5.tar.gz
tar -xvf drbd-8.2.5.tar.gz
cd /root/drbd-8.2.5

We need to build the DRBD driver and specify the path to the kernel source, then install the driver in the /lib path:

make KDIR=/usr/src/linux-source-2.6.24
make install

Once the driver is compiled, we need to move/copy it to the appropriate lib directory:

mv /lib/modules/2.6.24.3/kernel/drivers/block/drbd.ko 
    /lib/modules/2.6.24-16-server/kernel/drivers/block

Next we need to start the driver and tell Linux to load it the next time it boots:

modprobe drbd
echo 'drbd' >> /etc/modules
update-rc.d drbd defaults

Now that everything is installed, verify the driver is loaded:

lsmod | grep drbd

It might be a good idea to reboot and make sure it loads.

At this point, you should set up the /etc/drbd.conf, which mine looks like this:

global {
  usage-count no;
}

common {
  protocol C;

  syncer {
    rate 30M;
    al-extents 1801;
  }

  startup {
    wfc-timeout  0;
    degr-wfc-timeout 15;
  }

  disk {

    on-io-error   detach;
    # fencing resource-and-stonith;
  }

  net {
    sndbuf-size 512k;
    timeout        60;   #  6 seconds  (unit = 0.1 seconds)
    connect-int    10;   # 10 seconds  (unit = 1 second)
    ping-int       10;   # 10 seconds  (unit = 1 second)
    ping-timeout   5;    # 500 ms (unit = 0.1 seconds)
    max-buffers    8000;
    max-epoch-size 8000;
    cram-hmac-alg  "sha1";
    shared-secret  "secret";
  }
}

resource r0 {
  on app1 {
    disk       /dev/md2;
    address    10.26.210.10:7788;
    device     /dev/drbd0;
    meta-disk  internal;
  }

  on app2 {
    disk      /dev/sda3;
    address   10.26.210.11:7788;
    device     /dev/drbd0;
    meta-disk  internal;
  }
}

Notice the disk is different for each machine. The first machine is a software raid (md) while the second is a single drive (sda). Your setup will most likely be /dev/sdaX if you are using SATA or a RAID card, but it could be /dev/hdaX. You can use cfdisk to quickly check your partitions.

With the configuration set, you need to restart/reload DRBD, create the meta disk, and bring the drive up:

/etc/init.d/drbd restart
drbdadm create-md r0
drbdadm up r0

At this point you can view the DRBD status:

chris@app1:~$ cat /proc/drbd
version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12
 0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
    ns:221871972 nr:7160 dw:3856764 dr:227396211 al:763 bm:17504 lo:0 pe:0 ua:0 ap:0
	resync: used:0/31 hits:13841287 misses:13628 starving:0 dirty:0 changed:13628
	act_log: used:0/1801 hits:961638 misses:790 starving:0 dirty:27 changed:763

chris@app1:~$ /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12
m:res  cs         st                 ds                 p  mounted    fstype
0:r0   Connected  Primary/Secondary  UpToDate/UpToDate  C  /mnt/data  ext3

You’re now ready to mount /dev/drbd0 and put data on it. There’s still a couple other things you need to do if you plan on using Heartbeat for failover monitoring.

Download the Driver

If you are running an amd64 architecture, you can download the already compiled driver that was built with the steps above. Just to be clear, this was compiled for Ubuntu 8.04 with the 2.6.24-16-server kernel.

Just put the file in the drivers folder and don’t forget to set the proper owner.

wget http://www.cb1inc.com/files/drbd.ko
chown root:root drbd.ko
mv drbd.ko /lib/modules/2.6.24-16-server/kernel/drivers/block

I hope this is of some help and good luck!


MinneBar 2008 This Weekend!

May 08, 2008

This Saturday, May 10th, is MinneBar, Minnesota’s BarCamp. MinneBar is described as an “(un)Conference” which means it’s a free, ad-hoc gathering of technology folks where everyone is encouraged to contribute.

MinneBar

There are a lot of great sessions this year. I’ll be giving a presentation titled “Memcached & MySQL Sitting in a Tree.” The talk is about the new Memcached Functions for MySQL. I’ll talk a bit about the what, why, and how about this set of awesome UDFs.

I’m not sure what time I present and I think I have 50 minutes, but I don’t know for sure. I’m trying something new this time around; I’ll be publishing my presentation on SlideShare.

We are still 3 days away and there are currently 356 people signed up which is right around how many people were signed up last year. If you are in the Minneapolis/St. Paul area, you should come to participate and learn!

To register, visit their website, click the “login” link in the top right, use the password “c4mp” to login, then edit the main page, and add yourself to the bottom. Registration starts at 8:00am, so remember to set an alarm. 🙂

Hope to see you there!