CB1, INC.

Here's my presentation I gave June 9, 2008, at the Twin Cities MySQL and PHP User Group about my highly available cluster using DRBD and Heartbeat.

I added a few slides and cleaned things up a bit. The presentation went well and we had a lot of good questions.

The MySQL and PHP User Group will be taking some time off over the summer. There will be another meetup mid-summer to come up with some ideas for future meetings.

DRBD is a block device driver for Linux that allows you to mirror a partition between two servers.

I had a single application server, but whenever a server failure occurred, my websites, Subversion repository, and e-mail would go down. In order to be highly available, I added a second server in the event of a failure.

I don't have a ton of load, so the second server doesn't have to be super powerful.

Each machine has 2 network cards: one for public and one for private traffic. The public interfaces connect to my main switch on the 192.168.0.X network. The private interfaces are connected via a crossover cable on the 10.26.210.X network.

Once you have installed Ubuntu 8.04 Server, you need to install some build tools. I don't know specifically which build packages you need, so I just install a bunch of them and it should work. :)

apt-get install build-essential binutils cpp gcc autoconf automake1.9 libtool \
autotools-dev g++ make flex

In theory, that should get all of the development tools installed that you'll need.

Next we need to get the entire kernel source code. Just the kernel headers won't cut it. We need to compile the kernel so that it builds some of the scripts needed to compile the DRBD driver. We'll also install the ncurses library so that menuconfig works.

apt-get install libncurses5-dev linux-source-2.6.24

Then extract the kernel source:

cd /usr/src
tar -xvf linux-source-2.6.24.tar.bz2
cd /usr/src/linux-source-2.6.24

Next, lets clean up any unneeded files (which there shouldn't be any the first time):

make mrproper

Before you can build the kernel, you need to copy your existing kernel build configuration into the kernel source directory:

cp /boot/config-2.6.24-16-server /usr/src/linux-source-2.6.24/.config

Now we run the menuconfig which will read in our kernel build configuration and build some version files. As soon as the GUI appears, just exit immediately. You don't have to change any of the settings.

make menuconfig

Finally we need to prepare the kernel and compile it. This will take quite some time.

make prepare
make

Now that we have the kernel source compiled and ready to go, let's get the DRBD source.

cd /root
wget http://oss.linbit.com/drbd/8.2/drbd-8.2.5.tar.gz
tar -xvf drbd-8.2.5.tar.gz
cd /root/drbd-8.2.5

We need to build the DRBD driver and specify the path to the kernel source, then install the driver in the /lib path:

make KDIR=/usr/src/linux-source-2.6.24
make install

Once the driver is compiled, we need to move/copy it to the appropriate lib directory:

mv /lib/modules/2.6.24.3/kernel/drivers/block/drbd.ko \
    /lib/modules/2.6.24-16-server/kernel/drivers/block

Next we need to start the driver and tell Linux to load it the next time it boots:

modprobe drbd
echo 'drbd' >> /etc/modules
update-rc.d drbd defaults

Now that everything is installed, verify the driver is loaded:

lsmod | grep drbd

It might be a good idea to reboot and make sure it loads.

At this point, you should set up the /etc/drbd.conf, which mine looks like this:

global {
  usage-count no;
}

common {
  protocol C;

  syncer {
    rate 30M;
    al-extents 1801;
  }

  startup {
    wfc-timeout  0;
    degr-wfc-timeout 15;
  }

  disk {

    on-io-error   detach;
    # fencing resource-and-stonith;
  }

  net {
    sndbuf-size 512k;
    timeout        60;   #  6 seconds  (unit = 0.1 seconds)
    connect-int    10;   # 10 seconds  (unit = 1 second)
    ping-int       10;   # 10 seconds  (unit = 1 second)
    ping-timeout   5;    # 500 ms (unit = 0.1 seconds)
    max-buffers    8000;
    max-epoch-size 8000;
    cram-hmac-alg  "sha1";
    shared-secret  "secret";
  }
}

resource r0 {
  on app1 {
    disk       /dev/md2;
    address    10.26.210.10:7788;
    device     /dev/drbd0;
    meta-disk  internal;
  }

  on app2 {
    disk      /dev/sda3;
    address   10.26.210.11:7788;
    device     /dev/drbd0;
    meta-disk  internal;
  }
}

Notice the disk is different for each machine. The first machine is a software raid (md) while the second is a single drive (sda). Your setup will most likely be /dev/sdaX if you are using SATA or a RAID card, but it could be /dev/hdaX. You can use cfdisk to quickly check your partitions.

With the configuration set, you need to restart/reload DRBD, create the meta disk, and bring the drive up:

/etc/init.d/drbd restart
drbdadm create-md r0
drbdadm up r0

At this point you can view the DRBD status:

chris@app1:~$ cat /proc/drbd
version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12
 0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
    ns:221871972 nr:7160 dw:3856764 dr:227396211 al:763 bm:17504 lo:0 pe:0 ua:0 ap:0
	resync: used:0/31 hits:13841287 misses:13628 starving:0 dirty:0 changed:13628
	act_log: used:0/1801 hits:961638 misses:790 starving:0 dirty:27 changed:763

chris@app1:~$ /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12
m:res  cs         st                 ds                 p  mounted    fstype
0:r0   Connected  Primary/Secondary  UpToDate/UpToDate  C  /mnt/data  ext3

You're now ready to mount /dev/drbd0 and put data on it. There's still a couple other things you need to do if you plan on using Heartbeat for failover monitoring.

Download the Driver

If you are running an amd64 architecture, you can download the already compiled driver that was built with the steps above. Just to be clear, this was compiled for Ubuntu 8.04 with the 2.6.24-16-server kernel.

• drbd.ko [2.9MB]

Just put the file in the drivers folder and don't forget to set the proper owner.

wget http://www.cb1inc.com/sites/default/files/drbd.ko
chown root:root drbd.ko
mv drbd.ko /lib/modules/2.6.24-16-server/kernel/drivers/block

I hope this is of some help and good luck!

The day of tutorials started out with All Bases Covered: A Hands-on Introduction to High-availability MySQL and DRBD by Florian Haas and Philipp Reisner.

After a brief introduction to DRBD, they started discussing the configuration file. There were a couple settings that I had set incorrectly on my servers.

Since I have my two servers connected via a gigabit crossover cable, I had my synchronization rate set to 125MB. They recommended approximately 1/3 your network and disk I/O so that you're applications don't freeze up during synchronization. Their test system used 30MB so I'll give it a try too.

Another setting they had different was the activity log extents. All of the references I looked at said to set the al-extents to 257. Actually, there's an equation to find this value which is E = (R x t) / 4 where E is the al-extents, R is the synchronization rate, and t is the target synchronization time (in seconds). If the sync rate is 30MB and target sync time is 240 seconds, then the extent would be 1800, which rounded to the nearest prime is 1801.

Heartbeat is the cluster manager to detect when a node is unavailable. You should have at least 2 heartbeat connections between the two nodes. If eth0 is your public network and eth1 is your private network, you will want to configure Heartbeat to send the heartbeats across the public network using multicast and broadcast for the private network.

# /etc/ha.d/ha.cf
bcast eth1
mcast eth0 239.0.0.42 694 1 0

The version of Heartbeat that they demonstrated was Heartbeat v2. I use the older v1, which isn't as powerful, but much simpler to configure. It was also the first time I have seen the Heartbeat GUI. The GUI makes it easy as cake to manage the Heartbeat resources and offers a level of monitoring. You can tell the GUI was written by a developer since the usability could be improved greatly.

I specifically asked if DRBD has any issues with partitions larger than 2TB and Florian basically said if you can create the partition (meaning the driver supports it), then DRBD supports it. He mentioned something about how all SCSI devices use 32-bit integers for addressing that limits you to 2TB. This was news to me. My SATA RAID card is technically seen by Linux as a SCSI device. I'm not sure if this is 100% accurate, but nevertheless there is an easy solution. If you have 4TB of space, you can chop it up into two 2TB partitions, then use either software RAID 0 (stripe) or LVM (linear or striped map mode).

I can't wait to build my next HA cluster, but this time using Heartbeat v2.