Here’s my presentation I gave June 9, 2008, at the Twin Cities MySQL and PHP User Group about my highly available cluster using DRBD and Heartbeat.

I added a few slides and cleaned things up a bit. The presentation went well and we had a lot of good questions.

The MySQL and PHP User Group will be taking some time off over the summer. There will be another meetup mid-summer to come up with some ideas for future meetings.


4 Comments

  1. Thanks, I appreciated the presentation you put together. I do have some questions, and I beg your forgiveness as I am not yet MySQL savvy.

    How does DRDB handle an in-flight transaction during a disaster? For instance, if a block is in the process of being written on the primary/active node when the power goes out, is the passive node negatively affected (assuming a different power source)? Is the data block only shipped to the passive node when the write is complete? More details about those “worse case scenarios” would be helpful in my understanding of the limitations of the DRDB approach, if any.

    Without LinuxHA (Heartbeat), what comparisons can be drawn between DRDB and a cronn’ed rsync job? I have a basic understanding that DRDB would be much faster; does it have any other superior qualities that would steer me away from rsync?

    I am also curious if this is the method you would use for a Drupal application. From what I have researched so far, MySQL Replication, as a scale-out solution, is not good for Drupal based on Drupal’s inability to be “replication aware”. Obviously, MySQL Cluster is the high-end, the Cadillac, of High Availability, but we are trying to stay away from those big dollars if we can help it.

    Thanks for your time,

    Comment by Charles Schultz — October 16, 2008 @ 6:52 pm

  2. If data is a top priority, you’re probably best off using InnoDB. So, if you are doing say a bulk insert into a table, then the power goes out, only a few of the blocks get replicated to the other server.

    When Heartbeat detects the primary node is down, the secondary becomes the primary and starts up MySQL. When MySQL loads, it automatically checks the InnoDB database to see if it’s corrupt or if any incomplete transactions need to be rolled back.

    So, can you lose data for DRBD? Yes. Can the InnoDB recovery process take minutes to complete? Yes. It is for these reasons that I don’t use DRBD for anything other than a development environment, email server, or file server.

    DRBD is better than rsync because it’s near real time and can be done while MySQL is running. With rsync, you’d have to flush all the tables to disk, lock them, do the rsync, then unlock the tables. This could be a lengthy process and I’m not sure that rsyncing the raw database files are a good idea. It would probably be better to do a backup and ship the backup file to the other server and that takes time too.

    If you have a Drupal site, DRBD is a great choice because A) Drupal uses MyISAM which is flushed to disk immediately (but can still corrupt: see REPAIR TABLE) and B) you don’t have to mess around with Drupal’s database.

    The best solution, in my opinion, for failover and data integrity is master-master replication with InnoDB tables because failover is near instantaneous and InnoDB protects you from incomplete transactions. With a Drupal site, you’re probably going to have to modify parts of Drupal’s database to get it to work. Replication is also a bit more work to set up and maintain.

    Hope this helps! Just let me know if you have more questions. :)

    Comment by Chris Barber — October 17, 2008 @ 3:52 pm

  3. Hello,

    I really appreciate the work that you have done in your presentation. Here is what I am attempting to do. We are on a limited budget so I am attempting to do it all on a “shoe-string”.

    I would like to know the options of using the drdb technology for just one application – my email server. I would like to set up one email server at our main site and another at an off site connected via our internal campus network. From what I understand looking at your presentation and reading the drdb documentation – I should be able to have it so if one server goes down the other server can step in and take over.

    Is this true? Do you know of any How-to’s to get this running. I kind of need a walk-through as this concept is really new to me.

    thanks again.

    Comment by Joseph Norris — December 3, 2008 @ 6:36 pm

  4. DRBD and Heartbeat would work perfectly for a mail server. You’ll want to use a mail server that stores each message to disk so that it can get replicated.

    Since the other server is off site, you may need to change the “protocol” setting to speed things up, but you may run the risk of data loss. Check out DRBD’s replication protocols for more information.

    Just so you know, when server A goes down, it can take a couple seconds for things to fail over. If you are just using a mail server, then it’ll be quick, but if you are also failing over Apache, MySQL, Samba, FTP, and other services, it can take longer.

    Next you’ll want the connection to be as fast as possible. Ideally gigabit would be great, but things would still work using a slower connection.

    Probably the most important thing is that you have two separate mechanisms that allow each server to talk to each other. This is to prevent split brain, which is bad. You never want 2 masters. Your best bet is to use two network cards so that if one network goes down, they are still talking despite the network that went down is causing a service outage. You need to make sure that neither network card is talking to the other server through a single device at any point along the network.

    To learn more about my setup and how to install DRBD and Heartbeat, read my post on Installing DRBD on Ubuntu. Even if you don’t use Ubuntu, the settings should still work.

    Good luck!

    Comment by Chris Barber — December 3, 2008 @ 11:46 pm

RSS feed for comments on this post. TrackBack URL

Sorry, the comment form is closed at this time.