Updated! 6/6/2008

I posted the amd64 version of the compiled DRBD 8.2.5 driver at the end of this post!

DRBD is a block device driver for Linux that allows you to mirror a partition between two servers.

I had a single application server, but whenever a server failure occurred, my websites, Subversion repository, and e-mail would go down. In order to be highly available, I added a second server in the event of a failure.

app1.cb1inc.com app2.cb1inc.com
AMD Opteron 180 2.4GHz dual-core
4GB RAM DDR
2 x 250GB SATA 7,200 hard drives (Software RAID 1)
2 x Gigabit network cards
AMD Sempron 2800+
3GB RAM DDR
250GB IDE 7,200 hard drive
2 x Gigabit network cards

I don’t have a ton of load, so the second server doesn’t have to be super powerful.

Each machine has 2 network cards: one for public and one for private traffic. The public interfaces connect to my main switch on the 192.168.0.X network. The private interfaces are connected via a crossover cable on the 10.26.210.X network.

Once you have installed Ubuntu 8.04 Server, you need to install some build tools. I don’t know specifically which build packages you need, so I just install a bunch of them and it should work. 🙂

apt-get install build-essential binutils cpp gcc autoconf automake1.9 libtool autotools-dev g++ make flex

In theory, that should get all of the development tools installed that you’ll need.

Next we need to get the entire kernel source code. Just the kernel headers won’t cut it. We need to compile the kernel so that it builds some of the scripts needed to compile the DRBD driver. We’ll also install the ncurses library so that menuconfig works.

apt-get install libncurses5-dev linux-source-2.6.24

Then extract the kernel source:

cd /usr/src
tar -xvf linux-source-2.6.24.tar.bz2
cd /usr/src/linux-source-2.6.24

Next, lets clean up any unneeded files (which there shouldn’t be any the first time):

make mrproper

Before you can build the kernel, you need to copy your existing kernel build configuration into the kernel source directory:

cp /boot/config-2.6.24-16-server /usr/src/linux-source-2.6.24/.config

Now we run the menuconfig which will read in our kernel build configuration and build some version files. As soon as the GUI appears, just exit immediately. You don’t have to change any of the settings.

make menuconfig

Finally we need to prepare the kernel and compile it. This will take quite some time.

make prepare
make

Now that we have the kernel source compiled and ready to go, let’s get the DRBD source.

cd /root
wget http://oss.linbit.com/drbd/8.2/drbd-8.2.5.tar.gz
tar -xvf drbd-8.2.5.tar.gz
cd /root/drbd-8.2.5

We need to build the DRBD driver and specify the path to the kernel source, then install the driver in the /lib path:

make KDIR=/usr/src/linux-source-2.6.24
make install

Once the driver is compiled, we need to move/copy it to the appropriate lib directory:

mv /lib/modules/2.6.24.3/kernel/drivers/block/drbd.ko 
    /lib/modules/2.6.24-16-server/kernel/drivers/block

Next we need to start the driver and tell Linux to load it the next time it boots:

modprobe drbd
echo 'drbd' >> /etc/modules
update-rc.d drbd defaults

Now that everything is installed, verify the driver is loaded:

lsmod | grep drbd

It might be a good idea to reboot and make sure it loads.

At this point, you should set up the /etc/drbd.conf, which mine looks like this:

global {
  usage-count no;
}

common {
  protocol C;

  syncer {
    rate 30M;
    al-extents 1801;
  }

  startup {
    wfc-timeout  0;
    degr-wfc-timeout 15;
  }

  disk {

    on-io-error   detach;
    # fencing resource-and-stonith;
  }

  net {
    sndbuf-size 512k;
    timeout        60;   #  6 seconds  (unit = 0.1 seconds)
    connect-int    10;   # 10 seconds  (unit = 1 second)
    ping-int       10;   # 10 seconds  (unit = 1 second)
    ping-timeout   5;    # 500 ms (unit = 0.1 seconds)
    max-buffers    8000;
    max-epoch-size 8000;
    cram-hmac-alg  "sha1";
    shared-secret  "secret";
  }
}

resource r0 {
  on app1 {
    disk       /dev/md2;
    address    10.26.210.10:7788;
    device     /dev/drbd0;
    meta-disk  internal;
  }

  on app2 {
    disk      /dev/sda3;
    address   10.26.210.11:7788;
    device     /dev/drbd0;
    meta-disk  internal;
  }
}

Notice the disk is different for each machine. The first machine is a software raid (md) while the second is a single drive (sda). Your setup will most likely be /dev/sdaX if you are using SATA or a RAID card, but it could be /dev/hdaX. You can use cfdisk to quickly check your partitions.

With the configuration set, you need to restart/reload DRBD, create the meta disk, and bring the drive up:

/etc/init.d/drbd restart
drbdadm create-md r0
drbdadm up r0

At this point you can view the DRBD status:

chris@app1:~$ cat /proc/drbd
version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12
 0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
    ns:221871972 nr:7160 dw:3856764 dr:227396211 al:763 bm:17504 lo:0 pe:0 ua:0 ap:0
	resync: used:0/31 hits:13841287 misses:13628 starving:0 dirty:0 changed:13628
	act_log: used:0/1801 hits:961638 misses:790 starving:0 dirty:27 changed:763

chris@app1:~$ /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12
m:res  cs         st                 ds                 p  mounted    fstype
0:r0   Connected  Primary/Secondary  UpToDate/UpToDate  C  /mnt/data  ext3

You’re now ready to mount /dev/drbd0 and put data on it. There’s still a couple other things you need to do if you plan on using Heartbeat for failover monitoring.

Download the Driver

If you are running an amd64 architecture, you can download the already compiled driver that was built with the steps above. Just to be clear, this was compiled for Ubuntu 8.04 with the 2.6.24-16-server kernel.

Just put the file in the drivers folder and don’t forget to set the proper owner.

wget http://www.cb1inc.com/files/drbd.ko
chown root:root drbd.ko
mv drbd.ko /lib/modules/2.6.24-16-server/kernel/drivers/block

I hope this is of some help and good luck!


15 Comments

  1. Hey,

    thanks for this guide. On my Hardy servers, I also had to add the “flex” package for drbd to compile: “apt-get install flex”.

    Also, the compiled module was put into /lib/modules/2.6.24.3/kernel/drivers/block. I had to manually copy it to the correct location, which in my case was /lib/modules/2.6.24-17-server/ubuntu/block/drbd.

    Comment by kbe — May 29, 2008 @ 7:39 pm

  2. During the make process for DRBD, I received an error as Flex was not installed.

    An apt-get install flex of course did the trick, but figured that if anyone else hits that problem and can’t figure out why, there’s the solution. =)

    Comment by Arun U. Kapil — May 30, 2008 @ 6:28 pm

  3. I verified the compiled driver is not being dropped into the appropriate lib directory, but for some reason it works for me. In any case, I’ve updated the procedure above to move the driver before installing it.

    Also, good find on flex missing! I’ve updated the procedure to include flex.

    Thank you!

    Comment by Chris Barber — May 30, 2008 @ 6:40 pm

  4. Thanks for the article. I’ve been working on getting 8.2.5 running on Ubuntu 8.04LTS. What I hadn’t done up until now was get the linux source. Whilst DRBD built OK with the headers only and appeared to install, I noticed that

    cat /proc/drbd

    gave a GIT-hash string which indicated a different version (8.0.11 I think). Although this seemed to communicate OK with a 8.2.5 module on an openSUSE I wanted to make sure the drbd module on Ubuntu was the same version.

    Just one slight correction to make. Shouldn’t

    cp /boot/config-2.6.24-16-server /usr/src/linux-source-2.6.24

    be

    cp /boot/config-2.6.24-16-server /usr/src/linux-source-2.6.24/.config

    ?

    When I tried it, make menuconfig complained that .config couldn’t be found

    Comment by Paul Walsh — June 6, 2008 @ 8:42 am

  5. Further to what I just said…

    drbd 8.2.5 will build and install without having to compile the kernel:

    download drbd-8.2.5.tar.gz into /tmp then:

    apt-get install build-essential linux-kernel-devel \
        linux-headers-2.6.24-16 linux-headers-2.6.24-16-server
    cd /usr/src
    ln -s linux-headers-2.6.24-16-server linux
    tar xzvf /tmp/drbd-8.2.5.tar.gz
    cd /usr/src/linux
    make mrporper
    cd /usr/src/drbd-8.2.5
    make clean all
    make install

    Then, as kbe mentioned:

    copy  /lib/modules/2.6.24-16-server/kernel/drivers/block/drbd.ko \
         to /lib/modules/2.6.24-16-server/ubuntu/block/drbd/drbd.ko

    Comment by Paul Walsh — June 6, 2008 @ 9:05 am

  6. When I first started out, I was trying to compile DRBD with just the kernal headers, but I kept getting an error that modpost did not exist. The only way I could figure out how to solve it was to compile modpost via compiling the kernel.

    But, I would like to try your method someday and see if I had maybe missed a step before.

    Thanks!

    Comment by Chris Barber — June 6, 2008 @ 7:31 pm

  7. Thanks! I updated the post. I guess I should have copy and pasted what I typed. 🙂

    Comment by Chris Barber — June 6, 2008 @ 7:16 pm

  8. Hi folks,

    you just need to install the drbd-utils from repository and load the drbd module in /etc/modules.conf. the drbd modules is included in the ubuntu kernel since 8.04.

    see http://ubuntuforums.org/archive/index.php/t-766171.html

    Regards

    Jens

    Comment by Jens Wagner — June 13, 2008 @ 12:28 pm

  9. Ubuntu 8.04 ships with DRBD 8.0.11.

    http://packages.ubuntu.com/hardy/drbd8-utils

    When I was running Ubuntu 7.10, I tried using the 8.0.3 version it shipped with and had several issues. After talking one of the guys who wrote DRBD, he highly recommended using the latest version.

    The latest version at the time I wrote this article was 8.2.5 (8.2.6 is now out!). There has been several bug fixes and improvements between 8.0.11 and 8.2.5.

    Comment by Chris Barber — June 13, 2008 @ 4:07 pm

  10. How do you get around the fact that drbd is now included with the ubuntu modules? Does it matter?

    Here’s what I did:

    Download latest drbd source from debian here – http://packages.debian.org/experimental/drbd8-source

    build drbd source pacakge. install drbd source package. build module using module-assistant. install debian package that m-a created.

    However – m-a created the module /lib/modules/2.6.24-19-server/kernel/extra/drbd.ko and ubuntu has the module /lib/modules/2.6.24-19-server/ubuntu/block/drbd/drbd.ko. Whenver I do modprobe drbd the Ubuntu version is installed. I need the latest version to properly upgrade drbd0.7 to 0.8.X

    For now I moved /lib/modules/2.6.24-19-server/kernel/extra/drbd.ko to /lib/modules/2.6.24-19-server/ubuntu/block/drbd/drbd.ko but that’s going to be over-written if I ever upgrade the Ubuntu modules.

    Right now I’m thinking build linux-ubuntu-modules-2.6.24 with the newer source, but it seems like there should be a better way to keep these from conflicting.

    Comment by Anonymous — June 26, 2008 @ 4:40 pm

  11. Just thought I’d note that as the above user mentions, all the work done here doesn’t actually get drbd 8.2.x running on Hardy. We can see from the output that the version running is 8.0.11:

    chris@app1:~$ cat /proc/drbd
    version: 8.0.11 (api:86/proto:86)

    the fix posted above does help, but is a workaround and not a solution. If you are happy using 8.0.11 with hardy, you can avoid compiling your own kernel, and jump straight to the editing of drbd.conf by installing ‘drbd8-utils’.

    Comment by NaughtyusMaximus — July 4, 2008 @ 11:01 pm

  12. I recently configured drbd 8.2.5 on two Ubuntu 8.04 LTS boxes, but I wanted to point out some pitfalls in these instructions.

    When installing, I found that you must not mount the DRBD partition that you create. DRBD spits out an error message if you try and make an internal meta disk on an already mounted partition (seems obvious in retrospect). So I left the DRBD partitions unmounted during the install process in order to avoid this error. You could simply umount and then take care of it though.

    Also, this article fails to mention that you need to zero out the first part of the DRBD partition before you can create the internal metadisk. I had to issue the following command to get the metadisk to install (where /dev/sda2 is my drbd partition):

    if=/dev/zero bs=1M count=1 of=/dev/sda2; sync

    Next, it wasn’t a straightforward process of simply bringing up the arrays after creating the metadisk as described by this step:


    /etc/init.d/drbd restart
    drbdadm create-md r0
    drbdadm up r0

    Instead, after installing and restarting DRBD I found that both arrays were secondary and not considered up to date. I had to play around a bit to get things to sync. I don’t remember the exact process (I may outline it later), but I remember disconnecting the array I wanted to be primary, then promoting it to primary, then reconnecting the array to get things synced up.

    Now I am a little unsure about fstab. I can’t set it up so that I can write to the DRBD partition after rebooting without making the status appear as diskless on my primary server.. still a lot to be answered before my drbd/heartbeat configuration could ever be working.

    Comment by curiousgeorge — January 21, 2009 @ 9:46 pm

  13. 2 Points from me 🙂

    1) You only have to zero out the Device if you already created a filesystem on the partition you want to use with DRBD.

    2) On a fresh installed two node DRBD Setup you have to type (on both nodes if your ressource is called “r0”):
    drbdadm create-md r0
    drbdadm attach r0
    drbdadm connect r0

    and then on the machine you wish to promote as Primary:
    drbdadm — –overwrite-data-of-peer primary r0

    Now you can mount and mkfs what ever you want 🙂

    Cheers,
    Jan

    Comment by Jan — January 22, 2009 @ 6:08 pm

  14. One more thing: nice article, but …


    At this point you can view the DRBD status:


    chris@app1:~$ cat /proc/drbd
    version: 8.0.11 (api:86/proto:86)
    GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12
    0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
    ns:221871972 nr:7160 dw:3856764 dr:227396211 al:763 bm:17504 lo:0 pe:0 ua:0 ap:0
    ...

    You’ve loaded the wrong DRBD Module, your Version number is 8.0.11 which is the buildin module of Ubuntu 8.04.

    You have to overwrite it in this location:

    /lib/modules/2.6.24-19-server/ubuntu/block/drbd/drbd.ko

    btw. I couldn’t compile it with Paul Walsh’s method, too. I got the same error with modpost.

    Comment by Jan — January 22, 2009 @ 8:00 pm

  15. Thanks for the interesting tutorial and discussion.

    Could someone suggest to the best way to use DRBD for a general purpose office server. Currently my servers run postfix/dovecot/samba PDC/openvpn/Apache and i dream of having a system whereby if one dies the other comes to the rescue!

    whats the best way to protect a whole system like this?

    Thanks in advance

    Comment by diazepam — April 5, 2009 @ 10:43 am

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.