So, I  am catching up a bit on the technical documentation. A week taken to play Skyrim combined with various other bits and pieces made this a little difficult.

On the bright side, there are a few new things that have been worked on so hopefully plenty of things to cover soon.

We manage a number of servers and all over the place and all of them require to be backed up. We also have a number of desktops all with mirrored disks also getting backed up.

I like things to be all nicely efficient and its annoying when one server / desktop runs out of space when another two (or ten) has plenty of space. We grew to dislike NFS particularly due to the single point of failure and there were few other options.

We had tried glusterfs a few years ago (think it was at version 1.3 or something) and there were various issues particularly around small files and configuration was an absolute nightmare.

With high hopes that version 3.2 was exactly what we were looking for, we set up three basic machines for testing

Previously, glusterfs required all the configuration to be completed manually and with text files. It also required a fairly detailed knowledge of what they called translators and a lot of tweaking and fiddling with parameters.

I am very happy to report that this is no longer the case with 3.2.

The three servers(cserver[1-3]) are running Ubuntu and was updated to Natty (11.10) to get access to glusterfs 3.2 (11.04 only had 3.0). One thing to bear in mind is that the glusterfs website seemed to only have the 64 bit version but Ubuntu 11.10 also has the 32bit version.

Installing the server part of glusterfs was simple and straightfoward

$ sudo aptitude install glusterfs-server

Once this was done all three servers (terminator is a godsend when doing these things across a number of servers), adding the servers into a “cluster” was easy enough.

shri@cserver1:~$ gluster peer probe cserver2
Probe successful

shri@cserver1:~$ sudo gluster peer probe cserver3
Probe successful

The thing to note is that these probe statements are two way. In other words, all three servers are now part of the same cluster.

shri@cserver3:~$ sudo gluster peer status
Number of Peers: 2

Number of Peers: 3

Hostname: cserver1
Uuid: 8fe63300-e227-4aec-81f3-69b33f894330
State: Peer in Cluster (Connected)

Hostname: cserver2
Uuid: 275ce612-2dd8-4e2a-8cc8-3115ad18c594
State: Peer in Cluster (Connected)

The thing is that if you type in an incorrect hostname, the probe will keep trying to connect to it. I haven’t left it running long enough to know if it every returns.

shri@cserver3:~# sudo gluster peer probe does-not-exist
^C
shri@cserver3:~# sudo gluster peer status
Number of Peers: 3

Hostname: cserver1
Uuid: 8fe63300-e227-4aec-81f3-69b33f894330
State: Peer in Cluster (Connected)

Hostname: cserver2
Uuid: 275ce612-2dd8-4e2a-8cc8-3115ad18c594
State: Peer in Cluster (Connected)

Hostname: does-not-exist
Uuid: 00000000-0000-0000-0000-000000000000
State: Establishing Connection (Disconnected)

Thankfully, removing host does-not-exist is simple enough

shri@cserver3:~# sudo gluster peer detach does-not-exist
Detach successful
shri@cserver3:~# sudo gluster peer status
Number of Peers: 2

Hostname: cserver1
Uuid: 8fe63300-e227-4aec-81f3-69b33f894330
State: Peer in Cluster (Connected)

Hostname: cserver2
Uuid: 275ce612-2dd8-4e2a-8cc8-3115ad18c594
State: Peer in Cluster (Connected)

Creating a volume is straightforward. There are a number of different types of volumes which you can find out from the documentation. In this particular instance, we are creating a distributed replicated

shri@cserver3:~$ sudo gluster volume create testvol replica 2 transport tcp cserver1:/gdata cserver2:/gdata
Creation of testvol has been successful
Please start the volume to access data

The reason I have not included cserver3 in here is that the volume needs a multiple of the replica number of bricks. In the case, the there needs to be a muliple of 2 number of bricks.

Additionally, you could use rdma instead of tcp if you are using infiniband

Starting the volume is simple enough

shri@cserver3:~$ sudo gluster volume start testvol

this volume is now accessible from all the boxes in the cluster

shri@cserver1:~# sudo gluster volume info

Volume Name: testvo
Type: Distributed-Replicate
Status: Started
Number of Bricks: 2 x 1 = 2
Transport-type: tcp
Bricks:
Brick1: cserver1:/gdata
Brick2: cserver2:/gdata

Mounting this from another box is easy

$ sudo aptitude install glusterfs-client
$ sudo mount -t glusterfs /mnt cserver:/testvol

If you get the error of “endpoint not connected” when listing the content of the mount, it is likely because the volume is not started.

If you are curious, check the gdata folders in the bricks after copying some files into the mount and you’ll find them show up intact and on both bricks in the above example.