Here’s my presentation I gave June 9, 2008, at the Twin Cities MySQL and PHP User Group about my highly available cluster using DRBD and Heartbeat.
I added a few slides and cleaned things up a bit. The presentation went well and we had a lot of good questions.
The MySQL and PHP User Group will be taking some time off over the summer. There will be another meetup mid-summer to come up with some ideas for future meetings.
I posted the amd64 version of the compiled DRBD 8.2.5 driver at the end of this post!
DRBD is a block device driver for Linux that allows you to mirror a partition between two servers.
I had a single application server, but whenever a server failure occurred, my websites, Subversion repository, and e-mail would go down. In order to be highly available, I added a second server in the event of a failure.
| app1.cb1inc.com | app2.cb1inc.com |
|---|---|
| AMD Opteron 180 2.4GHz dual-core 4GB RAM DDR 2 x 250GB SATA 7,200 hard drives (Software RAID 1) 2 x Gigabit network cards |
AMD Sempron 2800+ 3GB RAM DDR 250GB IDE 7,200 hard drive 2 x Gigabit network cards |
I don’t have a ton of load, so the second server doesn’t have to be super powerful.
Each machine has 2 network cards: one for public and one for private traffic. The public interfaces connect to my main switch on the 192.168.0.X network. The private interfaces are connected via a crossover cable on the 10.26.210.X network.
Once you have installed Ubuntu 8.04 Server, you need to install some build tools. I don’t know specifically which build packages you need, so I just install a bunch of them and it should work. 
apt-get install build-essential binutils cpp gcc autoconf automake1.9 libtool autotools-dev g++ make flex
In theory, that should get all of the development tools installed that you’ll need.
Next we need to get the entire kernel source code. Just the kernel headers won’t cut it. We need to compile the kernel so that it builds some of the scripts needed to compile the DRBD driver. We’ll also install the ncurses library so that menuconfig works.
apt-get install libncurses5-dev linux-source-2.6.24
Then extract the kernel source:
cd /usr/src tar -xvf linux-source-2.6.24.tar.bz2 cd /usr/src/linux-source-2.6.24
Next, lets clean up any unneeded files (which there shouldn’t be any the first time):
make mrproper
Before you can build the kernel, you need to copy your existing kernel build configuration into the kernel source directory:
cp /boot/config-2.6.24-16-server /usr/src/linux-source-2.6.24/.config
Now we run the menuconfig which will read in our kernel build configuration and build some version files. As soon as the GUI appears, just exit immediately. You don’t have to change any of the settings.
make menuconfig
Finally we need to prepare the kernel and compile it. This will take quite some time.
make prepare make
Now that we have the kernel source compiled and ready to go, let’s get the DRBD source.
cd /root wget http://oss.linbit.com/drbd/8.2/drbd-8.2.5.tar.gz tar -xvf drbd-8.2.5.tar.gz cd /root/drbd-8.2.5
We need to build the DRBD driver and specify the path to the kernel source, then install the driver in the /lib path:
make KDIR=/usr/src/linux-source-2.6.24 make install
Once the driver is compiled, we need to move/copy it to the appropriate lib directory:
mv /lib/modules/2.6.24.3/kernel/drivers/block/drbd.ko
/lib/modules/2.6.24-16-server/kernel/drivers/block
Next we need to start the driver and tell Linux to load it the next time it boots:
modprobe drbd echo 'drbd' >> /etc/modules update-rc.d drbd defaults
Now that everything is installed, verify the driver is loaded:
lsmod | grep drbd
It might be a good idea to reboot and make sure it loads.
At this point, you should set up the /etc/drbd.conf, which mine looks like this:
global {
usage-count no;
}
common {
protocol C;
syncer {
rate 30M;
al-extents 1801;
}
startup {
wfc-timeout 0;
degr-wfc-timeout 15;
}
disk {
on-io-error detach;
# fencing resource-and-stonith;
}
net {
sndbuf-size 512k;
timeout 60; # 6 seconds (unit = 0.1 seconds)
connect-int 10; # 10 seconds (unit = 1 second)
ping-int 10; # 10 seconds (unit = 1 second)
ping-timeout 5; # 500 ms (unit = 0.1 seconds)
max-buffers 8000;
max-epoch-size 8000;
cram-hmac-alg "sha1";
shared-secret "secret";
}
}
resource r0 {
on app1 {
disk /dev/md2;
address 10.26.210.10:7788;
device /dev/drbd0;
meta-disk internal;
}
on app2 {
disk /dev/sda3;
address 10.26.210.11:7788;
device /dev/drbd0;
meta-disk internal;
}
}
Notice the disk is different for each machine. The first machine is a software raid (md) while the second is a single drive (sda). Your setup will most likely be /dev/sdaX if you are using SATA or a RAID card, but it could be /dev/hdaX. You can use cfdisk to quickly check your partitions.
With the configuration set, you need to restart/reload DRBD, create the meta disk, and bring the drive up:
/etc/init.d/drbd restart drbdadm create-md r0 drbdadm up r0
At this point you can view the DRBD status:
chris@app1:~$ cat /proc/drbd
version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12
0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
ns:221871972 nr:7160 dw:3856764 dr:227396211 al:763 bm:17504 lo:0 pe:0 ua:0 ap:0
resync: used:0/31 hits:13841287 misses:13628 starving:0 dirty:0 changed:13628
act_log: used:0/1801 hits:961638 misses:790 starving:0 dirty:27 changed:763
chris@app1:~$ /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12
m:res cs st ds p mounted fstype
0:r0 Connected Primary/Secondary UpToDate/UpToDate C /mnt/data ext3
You’re now ready to mount /dev/drbd0 and put data on it. There’s still a couple other things you need to do if you plan on using Heartbeat for failover monitoring.
If you are running an amd64 architecture, you can download the already compiled driver that was built with the steps above. Just to be clear, this was compiled for Ubuntu 8.04 with the 2.6.24-16-server kernel.
Just put the file in the drivers folder and don’t forget to set the proper owner.
wget http://www.cb1inc.com/files/drbd.ko chown root:root drbd.ko mv drbd.ko /lib/modules/2.6.24-16-server/kernel/drivers/block
I hope this is of some help and good luck!
Apache has a neat module called mod_dbd that allows your Apache modules to connect to a database. mod_dbd interfaces with apr_dbd, an Apache Portable Runtime (APR) abstraction layer around database specific drivers.
Back when Ubuntu 7.04 (fiesty) was released, a MySQL driver was not bundled with Apache for licensing concerns. So, in order to use mod_dbd to connect to a MySQL database, you need to get the MySQL driver source code from WebThing (apr_dbd_mysql.c) and manually re-compile apr-utils.
You also need the source code for Apache 2.2.3 (which includes apr-utils 1.2.7) from the Ubuntu 7.04 repositories, then copy the apr_dbd_mysql.c file into the Apache source apr-utils/dbd directory. The Ubuntu guys made a nice INSTALL.MySQL file in the apr-utils with some basic instructions.
What they don’t tell you is you need to install the MySQL source. To make matters worse, once you install it, the apr-utils 1.2.7 configure script can’t find it, even if you tell it where it is.
<snip> configure: checking for mysql in /usr/src/mysql-dfsg-5.0-5.0.38/include checking mysql.h usability... no checking mysql.h presence... no checking for mysql.h... no <snip>
This apparently was a known issue and was fixed in apr-utils 1.2.8.
Starting with apr-utils 1.2.11, the MySQL driver is bundled with it. Unfortunately, even Ubuntu 7.10 (gutsy) still ships with apr-utils 1.2.7. So, you are forced to download the source and compile.
Or, you can wait a couple days and Ubuntu 8.04 (hardy) which has Apache 2.2.8 and apr-utils 1.2.11. In theory the MySQL driver will work out of the box.
As for me, I’ll be compiling Apache, PHP, MySQL, memcached, and <insert essential infrastructure software> from source like I should have done in the beginning.
The Apache HTTP Web Server is a powerful and extensible web server that is the “A” in “LAMP”. One of the neat things about Apache is its API for writing custom modules.
Nick Kew wrote an excellent book called The Apache Modules Book. Anyone who is serious about Apache module development must buy this book.
Modules can be written a number of ways, but the most common way is to use the C programming language. For an C/C++ development IDE, I use KDevelop. It is pretty easy to use once you figure out what you need to do.
It is possible to write modules in C++, but I don’t recommend it if your module’s source can’t be contained in a single source file. There’s all sorts of interesting issues with exported symbols and static function declarations. Another reason to stick with C is pretty much all core modules and examples are written using C. You may give it a try and determine that it works just fine for your project.
Before you begin, there is a handful of applications and libraries you must have installed:
Launch KDevelop and select “New Project” from the Project menu. Since we are focusing on using C, select “Simple Hello world program” under the “C” folder. Give your module an “Application name” and specify the location to create the project.
On the next page of the wizard, you must enter your name, but your email address is not required.
The next couple wizard screens ask about version control and source templates. After finishing the wizard, you will be back at the IDE with the new project created.
Delete all of the source code that the editor created. Next paste the following code which originated from Nick’s version on The Apache Modules Book Companion site.
#include <httpd.h>
#include <http_protocol.h>
#include <http_config.h>
static int helloworld_handler(request_rec* r)
{
if (!r->handler || strcmp(r->handler, "helloworld"))
return DECLINED;
if (r->method_number != M_GET)
return HTTP_METHOD_NOT_ALLOWED;
ap_set_content_type(r, "text/html;charset=ascii");
ap_rputs("<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">n", r);
ap_rputs("<html><head><title>Hello World!</title></head>;", r);
ap_rputs("<body><h1>Hello World!</h1></body></html>", r);
return OK;
}
static void register_hooks(apr_pool_t* pool)
{
ap_hook_handler(helloworld_handler, NULL, NULL, APR_HOOK_MIDDLE);
}
module AP_MODULE_DECLARE_DATA helloworld_module = {
STANDARD20_MODULE_STUFF,
NULL,
NULL,
NULL,
NULL,
NULL,
register_hooks
};
Next we need to reconfigure the build target to create a library instead of a normal program. Right-click the build target and click “Remove”:
When the dialog displays, uncheck the “Also remove from disk” option before clicking “OK”. Now we need to add a new build target. Right-click the “src” folder and select “Add Target”:
From the “Add Target” dialog, change the type to “Libtool Library” and enter the name of the module. Also check the “-avoid-version” and “-module” options.
As soon as the target is created, right-click on it and make sure the checked options saved properly. Right-click on the target again and select “Make Target Active”:
Edit the project’s options by right-clicking the “src” folder and selecting “Options”:
Since this is a C project, we want to add the following options to the “CFLAGS” field:
-DLINUX=2 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_REENTRANT -pthread
From the “Includes” tab, add the following outside include directories:
Reorder the include paths so that “$(all_includes)” is first. Due to a bug or poor design, you must edit the two paths you just added and prepend a “-I”:
Since we deleted the old build target, we need to add the source files to the target by right-clicking the target and selecting “Add Existing Files”:
From the dialog, drag and drop the files you want to be apart of the target. For this simple example, we only move the “mod_helloworld.c” file.
We are all set to compile the project. From the “Build” menu, select “Build Active Target”:
If this is the first time you are performing the build, KDevelop will prompt you whether or not you want to run automake. Click the “Run Them” button to continue.
When the build is finished, the “Messages” panel will show up and display the build results. If everything went as planned, the output will say the build was successful.
The build process put the shared library file in the following location (assuming debug build):
/path/to/mod_helloworld/debug/src/.libs/libmod_helloworld
We need to install that file in the Apache modules directory which on Ubuntu is:
/usr/lib/apache2/modules
From a terminal, run the following command as root or sudo:
cp /path/to/mod_helloworld/debug/src/.libs/libmod_helloworld /usr/lib/apache2/modules/mod_helloworld.so
Next you’ll need to edit the Apache configuration file. In Ubuntu, the file is located at:
/etc/apache2/apache2.conf
You need to add the LoadModule and <Location> directives so Apache knows when to invoke the module.
LoadModule helloworld_module /usr/lib/apache2/modules/mod_helloworld.so
<Location /helloworld>
SetHandler helloworld
</Location>
I’ve had spotty luck where to actually insert those settings. After the LogLevel, but before any other LoadModule entries seems to work for me. After you save the changes, restart Apache using the following command as root or sudo:
apache2ctl restart
If your Apache acts funny, try restarting it again.
The last step is to test the module. Open up your favorite web browser and hit http://localhost/helloworld:
If everything worked, you should see something similar to the image above.
We have only scratched the surface. The Apache Portable Runtime (APR) provides a ton of functionality that makes developing modules much easier. The Apache Modules Book dives into several topics such as configuration settings, content generators, filters, and database connectivity.
Another great resource is Apache’s own module source code in their Subversion repository: http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/.
Day 1 of the Ubuntu Live conference is over and I can’t believe how worthless the sessions were. The keynotes and networking was great, but the sessions just sucked.
The whole day was kicked off by the main guy at Ubuntu, Mark Shuttleworth. According to Mark, we can expect another LTS (long-term support) release of Ubuntu around April 2008. Eben Moglen also gave an awesome keynote and it’s absolutely great to hear him speak about his thoughts of Redmond. I have to say that the keynotes, which typically put me to sleep, were really quite good. The speakers were funny, energetic, and interesting.
I don’t think I learned a thing. It’s either the content was way to high level or they never really got around to what the session was about. I think part of the problem was 30 minute sessions. This should have been closer to 45-50 minutes instead. All in all, the sessions were a big let down.
But not all sucked. I got to meet Mark Shuttleworth, which was cool. I also got to personally meet Eben Moglen, the legal genius behind GPL3. I met lots of cool attendees from all over: Wired, Ars Technica, Google, and many more. I got to hold a OLPC, which was pretty cool. The most memorable moment of the day was meeting Mårten Mickos, the CEO of MySQL AB. He was extremely approachable and great to talk to. He didn’t have one of those big CEO heads. Just a great guy and I look forward to the next time our paths cross.
About 2 months ago, I wrote that I was heading to Portland for the O’Reilly Open Source Conference. It turns out that O’Reilly is also hosting another conference about Ubuntu, a very popular Linux distro.
When I signed up for OSCON, I elected to go to the tutorials, which consists for 4 tutorials spanned across 2 days. Well, 2 of the 4 tutorials I signed up for were canceled. Needless to say, I’m pretty bummed out. But O’Reilly was nice enough to get me into the Ubuntu Live Sunday, July 22nd, 2007.
So, here’s the plan thus far for the first part of the week:
| 8:00am – 9:00am | Breakfast |
| 9:00am – 10:30am | Keynote: Mark Shuttleworth |
| Keynote: Stephen O’Grady | |
| Keynote: Jeff Waugh | |
| 10:30am – 11:00am | Break |
| 11:00am – 11:30am | Managing, maintaining, and securing Ubuntu machines |
| 11:30am – 11:35am | Break |
| 11:35am – 12:05pm | The 3 laws of IT and how open source can benefit |
| 12:05pm – 12:10pm | Break |
| 12:10pm – 12:40pm | The bleeding edge |
| 12:40pm – 2:00pm | Lunch |
| 2:00pm – 3:00pm | Keynote: Eben Moglen |
| Keynote: Mitchell Kapor | |
| Keynote: Jim Zemlin | |
| 3:00pm – 3:30pm | Break |
| 3:30pm – 4:00pm | Getting started with virtualization on Ubuntu |
| 4:00pm – 4:05pm | Break |
| 4:05pm – 4:35pm | Providing Your Software to Ubuntu Users |
| 4:35pm – 4:40pm | Break |
| 4:40pm – 5:10pm | Case studies: building virtual machines |
| 5:10pm – 5:15pm | Break |
| 5:15pm – 5:45pm | Success with Desktop Linux: Making Operating System Choice Irrelevant |
| 6:00pm – 8:00pm | Food and fun |
| 7:30am – 8:30am | Breakfast |
| 8:30am – 12:00pm | Pthreads programming |
| 12:00pm – 1:30pm | Lunch |
| 1:30pm – 5:00pm | Linux performance monitoring |
| 7:30am – 8:30am | Breakfast |
| 8:30am – 12:00pm | New parallel programming tools for a multicore world |
| 12:00pm – 1:30pm | Lunch |
| 1:30pm – 5:00pm | Creating a Linux based software appliance for SMBs |
| 5:00pm – 6:00pm | Break |
| 6:00pm – 7:00pm | Powell’s technical books events |
| 7:30pm – 10:00pm | Meet ‘n Geek |
| 8:00pm – 8:30pm | Google O’Reilly Open Source Awards |
In the spare time, I plan to hit up the Expo and see if there is anything good. I’m going to follow this plan, but if something else better comes up, I’ll change things up a bit. Stay tuned for more coverage of both the Ubuntu Live Conference and the Open Source Conference!
There are circumstances where you may need to have more than one version of Java installed, but how do you switch between them?
Execute the following command to list the installed JVMs:
sudo update-alternatives --config java
Which will output something similar to the following:
There are 3 alternatives which provide `java'.
Selection Alternative
-----------------------------------------------
1 /usr/bin/gij-wrapper-4.1
* 2 /usr/lib/jvm/java-1.5.0-sun/jre/bin/java
+ 3 /usr/lib/j2se/1.4/bin/java
Press enter to keep the default[*], or type selection number:
Simply enter the number for the JVM to use. The * shows the current selected JVM.
Thursday, April 19th, the latest rev of the Linux distro Ubuntu 7.04 (Feisty Fawn) was released. I wasted no time to download it and installing it in one of my trusty virtual machines.
I have a Macbook Pro and I use Parallels Desktop for Mac as my VM software. Installing previous Ubuntu releases was a piece of cake, but Ubuntu 7.04 was being a little “feisty” to install.
After trying the DVD, Desktop, and Alternative versions, I still had no luck. It was hard to tell what the problem was on the DVD and Desktop editions since they were live CDs and the screen just went blank. The Alternative version gave me a clue that the installer couldn’t load the CD/DVD drivers.
After doing some digging, I discovered there is a problem with both Parallels CD/DVD driver and the latest Linux kernel that ships with 7.04.
To get around the problem, I changed the OS Type to “Solaris” and OS Version to “Other Solaris” and then tried again to install Ubuntu.
Ubuntu’s installer found the drive without a problem and successfully installed the operating system. After the installation was complete, I shutdown the VM and changed the OS Type back to “Linux” and OS Version to “Other Linux kernel 2.6″. I have no idea if the CD/DVD drive works after changing these back, nor do I know if there are any issues with just leaving the OS Type as Solaris.
My hopes are that either a new kernel and/or Parallels releases an update to fix this problem. Honestly though, I have never used the CD/DVD drive in a Linux VM since everything I install is downloaded.
Has anybody else had this issue or come up with a better workaround?