DRBD8 with two primaries on debian etch

December 10th, 2008 | Tags: , , , , ,

The goal of this post is to set up an architecture composed of two nodes that can both simultaneously have read/write access to a synchronized partition. Each node will access the synchronized partition as any other classical partition, the only difference is that every modification will also be forwarded to the other node.

DRBD8 configuration file

System requirements

  • 2.6.24 kernel at least

[tux]# apt-get update
[tux]# apt-get install linux-image-2.6.24-etchnhalf.1-686
[tux]# apt-get install linux-headers-2.6.24-etchnhalf.1-686
[tux]# reboot

Configuration requirements

  • Adjust hostname, that is to say specify the same hostname as in /etc/drbd.conf in /etc/hostname.
  • Every resource has to be accessible from its hostname, that’s why you have to define every hostname in hosts section:

[tux]# vi /etc/hosts
192.168.9.xx hostname1.domain.org hostname1
192.168.9.xx hostname2.domain.org hostname2

Update your source list to be able to get packages from debian backports

  • Add the following line to your /etc/apt/sources.list:

deb http://www.backports.org/debian etch-backports main contrib non-free

  • If you are using etch and you want apt to verify the downloaded backports you can import backports.org archive’s key into apt then run apt-get update:

[tux]# apt-get install debian-backports-keyring
[tux]# apt-get update

Install DRBD8

  • Install DRBD8 source and DRBD8 utils:

[tux]# apt-get install drbd8-source drbd8-utils

  • Build DRBD8 module from previously installed sources using the module-assistant:

[tux]# module-assistant auto-install drbd8

Prepare the partition that will be synchronized thanks to DRBD

  • Generally it’s better to think about the DRBD partition earlier and prepare a dedicated partition during operating system installation. Since my machines are hosted on VMWare, I simply add a new virtual disk. Then we will process to its configuration. Do the same on both nodes as usual!
[tux]# fdisk -l

Disk /dev/sda: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device       Boot   Start    End     Blocks    Id                 System
/dev/sda1       *       1    743    5968116    83                  Linux
/dev/sda2             744    783     321300     5               Extended
/dev/sda5             744    783    321268+    82   Linux swap / Solaris

Disk /dev/sdb: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn't contain a valid partition table
  • Let’s configure the new disk added entitled /dev/sdb. First, create a logical partition on an extended one that will completely fill the new partition:
[tux]# fdisk /dev/sdb

Command (m for help): n
Command action
e   extended
p   primary partition (1-4)
e
Partition number (1-4): 1
First cylinder (1-130, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-130, default 130):
Using default value 130

Command (m for help): n
Command action
l   logical (5 or over)
p   primary partition (1-4)

l
First cylinder (1-130, default 1):
Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-130, default 130):

Using default value 130

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
  • In our case, the logical partition that will be synchronized by DRBD is entitled /dev/sdb5:
[tux]# fdisk -l /dev/sdb

Disk /dev/sdb: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device       Boot   Start    End     Blocks    Id      System
/dev/sdb1               1    130   1044193+     5    Extended
/dev/sdb5               1    130    1044162    83       Linux
  • /dev/sdb5 is ready to be used by DRBD. Thanks to the comment of Mark, we know that if there is no existing data on the device, we should not create a filesystem on the underlying device! It will be done later on the DRBD device itself.

Configure DRBD8

  • Adjust drbd.conf according to your system, that is to say specify your local partition as well as the correct hostnames and IP addresses, then copy it in /etc directory.
on hostname1 {
  device /dev/drbd0;
  disk /dev/sdb5;
  address 192.168.9.xx:7789;
  meta-disk internal;
}
on hostname2 {
  device /dev/drbd0;
  disk /dev/sdb5;
  address 192.168.9.xx:7789;
  meta-disk internal;
}

Create DRBD resource

  • First of all create your resource. Resource name corresponds to what you specified in your drbd.conf (here the resource name is r0). The DRBD partition must have the same size on both nodes to work correctly.
  • If there is no filesystem on /dev/sdb5, following commands should do the trick (on both nodes):
[tux]# drbdadm create-md r0
You want me to create a v08 style flexible-size internal meta data block.
There apears to be a v08 flexible-size internal meta data block
already in place on /dev/sdb5 at byte offset 106917888
Do you really want to overwrite the existing v08 meta-data?
[need to type 'yes' to confirm] yes

Writing meta data...
initialising activity log
NOT initialized bitmap
New drbd meta data block sucessfully created.
  • If any error like the following one occurs:
[tux]# drbdadm create-md r0
md_offset 1069215744
al_offset 1069182976
bm_offset 1069150208

Found ext3 filesystem which uses 1044160 kB
current configuration leaves usable 1044092 kB

Device size would be truncated, which
would corrupt data and result in
'access beyond end of device' errors.
You need to either
* use external meta data (recommended)
* shrink that filesystem first
* zero out the device (destroy the filesystem)
Operation refused.

Command 'drbdmeta /dev/drbd0 v08 /dev/sdb5 internal create-md'
terminated with exit code 40
drbdadm aborting
  • It means there was some data on /dev/sdb5 or at least a filesystem. In this kind of scenario there are two ways to solve the problem: adjust partition size or backup and erase existing data.
    1. Adjust partition size is the most dangerous option! Use it at your own risks. Many people encountered ‘accessing beyond end of device’ errors when the device was almost full! (thank you for your comment Mark ;-)). If you still want to do it,  use the current configuration leaves usable information returned by previous command (1044092) to resize the device. Please perform the same commands on each node.

[tux]# e2fsck -f /dev/sdb5 && resize2fs /dev/sdb5 1044092K

  1. The second option is the safer one: backup up existing data on another device, then erase its current content:

[tux]# dd if=/dev/zero of=/dev/sdb5

  • And then you can successfully create your resource on both machines:

[tux]# drbdadm create-md r0

Start DRBD8 and ensure it works correctly

  • You can start DRBD this way (will produce an error since at first startup you have to force only one node into primary state):

[hostname1]# /etc/init.d/drbd start
[hostname2]# /etc/init.d/drbd start

  • Or make it manually (attach and connect the resource r0):

[hostname1]# drbdadm attach r0
[hostname1]# drbdadm connect r0
[hostname2]# drbdadm attach r0
[hostname2]# drbdadm connect r0

  • In order to get consistent data, force DRBD partition into primary state for only one of the two nodes, this will trigger the synchronization process. (“drbdadm — –overwrite-data-of-peer primary r0″ is equivalent to the following command)

[hostame1]# drbdsetup /dev/drbd0 primary -o

  • Wait the end of the synchronization.
[hostname1]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.0.13 (api:86/proto:86)
GIT-hash: ee3ad77563d2e87171a3da17cc002ddfd1677dbe build by phil@fat-tyre,
2008-08-04 15:28:07
m:res  cs         st                ds                    p mounted fstype
...    sync'ed:   20.8%             (830012/1044092)K
0:r0   SyncSource Primary/Secondary UpToDate/Inconsistent C
  • When 100% is reached, you can successfully set the secondary node into primary state in order to get two primaries:

[hostame2]# drbdadm primary r0

  • Verify that everything works fine:
[tux]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.0.13 (api:86/proto:86)
GIT-hash: ee3ad77563d2e87171a3da17cc002ddfd1677dbe build by phil@fat-tyre,
2008-08-04 15:28:07
m:res cs        st              ds                p mounted fstype
0:r0  Connected Primary/Primary UpToDate/UpToDate C
  • Now it’s time to create your EXT3 filesystem on the DRBD device! (only in the case there was no existing data on /dev/sdb5)

[hostname1]# mke2fs -j /dev/drbd0

  • You can finally check if your system works correctly. First create a new directory:

[hostname1]# mkdir /synchronized
[hostname2]# mkdir /synchronized

  • Then mount the DRBD partition at this location. You should be able to access it read/write on both machines simultaneously.

[hostname1]# mount /dev/drbd0 /synchronized
[hostname2]# mount /dev/drbd0 /synchronized

  • If you were unable to mount your partition and get the following error message, ensure you have correctly created a valid filesystem on the DRBD device as explained before.

[tux] # mount /dev/drbd0 /synchronized/
mount: you must specify the filesystem type

  • If you followed every step carefully, your system must be set up correctly and the DRBD status command should return the following state on both nodes. You can see that DRBD is mounted on /synchronized.
[hostname1]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.0.13 (api:86/proto:86)
GIT-hash: ee3ad77563d2e87171a3da17cc002ddfd1677dbe build by phil@fat-tyre,
2008-08-04 15:28:07
m:res cs        st              ds                p mounted       fstype
0:r0  Connected Primary/Primary UpToDate/UpToDate C /synchronized ext3
  • According to DRBD specifications, our system and configuration is valid. We can try to create files from both locations to see if it works how it should.

[hostname1]# touch /synchronized/from_host1
[hostname2]# touch /synchronized/from_host2
[hostname1]# ls /synchronized
lost+found from_host1
[hostname2]# ls /synchronized
lost+found from_host2

  • As you can notice, our system does not work as expected, since we are not able to see the file added from the other node. I did not find a way to make it work properly using a standard Ext3 filesystem. Generally, you have to unmount the partition, put it into secondary state, put it back into primary state and mount it again to be sure to see the modifications of the other node. DRBD documentation specifies that in order to use the software safely with two primaries we have to couple DRBD with a specific filesystem that ensures integrity. I will go further and asserts that DRBD cannot work properly without a specific filesystem even if there is no corruption risk.

Other considerations

You have to keep in mind that there is no integrity guaranty between the two machines. Therefore we are in any case force to implement a lock mechanism to avoid data corruption. DRBD8 deals only with synchronization between nodes. In order to circumvent this drawback, the DRBD synchronized partition must use a specific file system like GFS2 for example if you want to see modifications from both nodes almost in real time. A next post will go into further details and explain how to build a shared nothing architecture between two nodes using DRBD8 + GFS2 on debian etch. I can already claim this time it will work as expected, so do not become discouraged.

  1. mgiunktode
    January 14th, 2009 at 17:42
    Reply | Quote | #1

    Thank you for this howto: It’s been very useful to me. Just an observation: you wrote “drbsetup” instead of “drbdsetup”.

    Thanks again!

  2. gcharriere
    January 14th, 2009 at 19:13
    Reply | Quote | #2

    @mgiunktode
    I’m glad that it could help you! I corrected the misprint…

  3. Mark
    March 13th, 2009 at 07:25
    Reply | Quote | #3

    NO!
    Never EVER create the filesystem on the underlying device!
    ALWAYS create your filesystem on the drbd device!

    both nodes:
    modprobe drbd
    drbdadm create-md ‘resourcename’
    drbdadm attach ‘resourcename’
    drbdadm connect ‘resourcename’

    on the PRIMARY node…
    drbdadm — –overwrite-data-of-peer primary ‘resourcename’
    ‘resourcename’ is the name of your resource, in this case ‘r0′
    If you do this any other way you WILL have ‘attempts to access beyond end of device’ errors.

    THEN.. when it’s all said and done…
    mke2fs -j /dev/drbd0

  4. March 13th, 2009 at 15:41
    Reply | Quote | #4

    @Mark
    I am forced to admit that what you explained is clearly a cleaner and safer solution than what I suggested with my resize2fs method. Thanks to your comment, I realized that in my original post, I proposed a special case that helps to deal with existing data.
    Thus I will enhance my post and mention your solution as the one to use if there is no existing data on the device.

  5. March 13th, 2009 at 17:16
    Reply | Quote | #5

    @gcharriere
    I modified the post to take into account Mark’s advices.

  6. alonso
    September 4th, 2009 at 11:20
    Reply | Quote | #6

    Thanks! It worked really nice.
    I’ve been looking for a long time for a decent howto on drbd… Until I’ve found yours! Very well detailed, just excelent!
    Thanks again!

  7. Tariq Ghouri
    July 3rd, 2010 at 12:44
    Reply | Quote | #7

    Marvellous document it soled my errors which has made me frustrated from last 2 days, thanks man.

  8. Goodies
    October 4th, 2010 at 15:04
    Reply | Quote | #8

    Thank You, if You need a good tutorial for a NFS and FTP implementation with DRBD and pacemaker visit http://docs.homelinux.org the tutorials on this site helped me a lot !

  9. February 24th, 2011 at 11:43
    Reply | Quote | #9

    Thanks! It worked really nice.
    I’ve been looking for a long time for a decent howto on drbd… Until I’ve found yours! Very well detailed, just excelent!
    Thanks again!

  10. ivan
    July 19th, 2012 at 09:12

    Thanks for your document! But I have following error when excuting the following on host2

    #drbdadm primary r0

    0: State change failed: (-1) Multiple primaries not allowed by config
    Command ‘drbdsetup 0 primary’ terminated with exit code 11

    Host 1’s status
    /etc/init.d/drbd status
    drbd driver loaded OK; device status:
    version: 8.3.7 (api:88/proto:86-91)
    GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@ubuntu, 2012-07-18 21:44:45
    m:res cs ro ds p mounted fstype
    0:r0 Connected Primary/Secondary UpToDate/UpToDate C /synchronized ext3

    Host 2’s status
    /etc/init.d/drbd status
    drbd driver loaded OK; device status:
    version: 8.3.7 (api:88/proto:86-91)
    GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@ubuntu, 2012-07-18 21:46:09
    m:res cs ro ds p mounted fstype
    0:r0 Connected Secondary/Primary UpToDate/UpToDate C

4 trackbacks

  1. DRBD8 + GFS2 on debian etch | HOWTO's and Tutorials Pingback | 2009/01/14
  2. LongVNIT » DRBD8 + GFS2 on debian etch Pingback | 2009/12/05
  3. look these up Trackback | 2014/08/17
  4. meilleurjeuhacks.wordpress.com Trackback | 2014/10/04
Comments are closed.