Debian - How to replicate data over network using GlusterFS

Recently I needed to replicate data between two Linux servers so I could mitigate potential hardware failure. To achieve this goal I used GlusterFS - an Open Source distributed file system. In this example I will use Debian Wheezy as I want to try more recent version of GlusterFS.

Assumptions

I will use server1 and server2 to create replicated volume. server3 and server4 to show how to extend it for size or replica count, server5 to migrate brick.

$ cat /etc/hosts | grep server
192.168.56.101 server1
192.168.56.102 server2
192.168.56.103 server3
192.168.56.104 server4
192.168.56.105 server5

I will use subdirectories of /var/export/ to store data internally and mirror /var/www directory.

Install GlusterFS

To use GlusterFS you need to install glusterfs-server package for server functionality and glusterfs-client (this package will be installed as dependency) to use created filesystem.

$ sudo apt-get install glusterfs-server

At the time of writing this post, the repository contains GlusterFS 3.2.7. You can skip it and install the latest version 3.3.0 using GlusterFS latest Debian packages link but you need to use Debian Wheezy because of libssl 1.0.0 dependency (Debian Squeeze contains libssl 0.9.8 version).

Create trusted storage pool

Create trusted network of storage servers, execute command on the first server:

server1# gluster peer probe server2

To add another server to trusted storage pool just probe it from any trusted server.

Verify peer status using command:

server1# gluster peer status
Number of Peers: 1

Hostname: server2
Uuid: aa251818-d4f6-47ab-a1d5-e9ced79d7daa
State: Peer in Cluster (Connected)

Create replicated volume between two servers

Create replicated volume www using two bricks. As stated above I will use directory /var/export/www on each server to store data internally.

# gluster volume create www replica 2 transport tcp server1:/var/export/www server2:/var/export/www

Set IP addresses of the clients which should be allowed to access this volume:

# gluster volume set www auth.allow 192.168.56.*

Start volume www:

# gluster volume start www

Check volume status using command:

# gluster volume info www
Volume Name: www
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: server1:/var/export/www
Brick2: server2:/var/export/www
Options Reconfigured:
auth.allow: 192.168.56.*

Mounting volume

To mount volume on each server use command:

# mount -t glusterfs localhost:/www /var/www

I used localhost hostname so if other server is down it will mount anyway.

To automatically mount volume at boot on each server add similar fstab entry:

$ cat /etc/fstab | grep glusterfs
/etc/glusterd/vols/www/www-fuse.vol /var/www glusterfs defaults,_netdev 0 0

Extend trusted storage pool

Before I could continue I need to add three servers to trusted storage pool:

# gluster peer probe server3
# gluster peer probe server4
# gluster peer probe server5

Verify peer status using command:

server1# gluster peer status
Number of Peers: 4

Hostname: server2
Uuid: aa251818-d4f6-47ab-a1d5-e9ced79d7daa
State: Peer in Cluster (Connected)

Hostname: server3
Uuid: cfab1973-69a6-4d43-b3e1-a4dea34e4bcc
State: Peer in Cluster (Connected)

Hostname: server4
Uuid: d9e342ac-f503-4311-d2b9-cb4dcea38391
State: Peer in Cluster (Connected)

Hostname: server5
Uuid: 4d0abff1-32a8-9dcc-8263-a7cd7d4ae9cf
State: Peer in Cluster (Connected)

Extend volume for size - replicated-distributed volume

To extend amount of disk space available on volume www add a brick on server3 and server4 (multiple of the replica count):

# gluster volume add-brick www server3:/var/export/www server4:/var/export/www

This is now a replicated-distributed volume (2x2) and twice of disk space is available

# gluster volume info www
Volume Name: www
Type: Replicate
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: server1:/var/export/www
Brick2: server2:/var/export/www
Brick3: server3:/var/export/www
Brick4: server4:/var/export/www
Options Reconfigured:
auth.allow: 192.168.56.*

Remove bricks

Remove bricks added in previous step. In case you created files on this volume you need to know that part of the data on will be lost.

# gluster volume remove-brick www server3:/var/export/www server4:/var/export/www

Remove directory /var/export/www on server3 and server4.

Extend volume for replica count

This is a bit tricky as for now there is no safe way to do it so backup is necessary.

Stop volume www:

# gluster volume stop www

Delete volume www:

# gluster volume delete www

Recreate volume but increase replica count to 4 and define all four bricks:

# gluster volume create www replica 4 transport tcp server1:/var/export/www server2:/var/export/www server3:/var/export/www server4:/var/export/www

Start volume www and set options:

# gluster volume start www
# gluster volume set www auth.allow 192.168.56.*

Check volume status:

# gluster volume info www
Volume Name: www
Type: Replicate
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: server1:/var/export/www
Brick2: server2:/var/export/www
Brick3: server3:/var/export/www
Brick4: server4:/var/export/www
Options Reconfigured:
auth.allow: 192.168.56.*

Volume data should become available again but I cannot guarantee that.

Migrate brick

Bricks can be easilly moved between servers. To move brick /var/export/www (volume www) from server4 to server5 execute commands below.

Start migration of server4:/var/export/www brick to server4:/var/export/www:

# gluster volume replace-brick server4:/var/export/www server5:/var/export/www start

Check status of above migration:

# gluster volume replace-brick server4:/var/export/www server5:/var/export/www status

After the migration is done you need to commit it:

# gluster volume replace-brick server4:/var/export/www server5:/var/export/www commit

Check volume status:

# gluster volume info www
Volume Name: www
Type: Replicate
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: server1:/var/export/www
Brick2: server2:/var/export/www
Brick3: server3:/var/export/www
Brick4: server5:/var/export/www
Options Reconfigured:
auth.allow: 192.168.56.*

GlusterFS Documentation

Milosz Galazka's Picture

About Milosz Galazka

Milosz is a Linux Foundation Certified Engineer working for a successful Polish company as a system administrator and a long time supporter of Free Software Foundation and Debian operating system.

Gdansk, Poland https://sleeplessbeastie.eu