Expanding glusterfs volumes [1112]

Once you have set up a glusterfs volume, you might want to expand the volume to add storage. This is an astoundingly easy task.

The first thing that you’ll want to do is to add in bricks. Bricks are similar to physical volumes a la LVM. The thing to bear in mind is that depending on what type of cluster you have (replicated / striped), you will need to add a certain number of blocks at a time.

Once you have a initialised the nodes, to add in a set of bricks, you need the following command which adds two more bricks to a cluster which keeps two replicas.

$ gluster volume add-brick testvol cserver3:/gdata cserver4:/gdata

Once you have done this, you will need to rebalance the cluster, which involves redistributing the files across all the bricks. There are two steps to this process, the “fixing” of the layout changes and the rebalancing of the data itself. You can perform both tasks together.

Continue reading

Tracking progress of an update statement [1101]

Sometimes there is a need to execute a long running update statement. This update statement might be modifying millions of rows as was the case when we went hunting for a way to track the progress of the update. Hunting around took us to http://archives.postgresql.org/pgsql-admin/2002-07/msg00286.php In our particular case, we are using postgresql but this should work with any database server that provides sequences. Our original sql was of the form:

update only table1 t1
set amount = t2.price
from table2 t2
where t1.id = t2.id;

There is of course now way of figuring out how many rows had been updated already. The first step was to create a sequence

CREATE TEMPORARY SEQUENCE seq_progress START 1;

We can then use this sequence in the update statement to ensure that each row updated also increments the sequence

update only table1 t1
set amount = t2.price
from table2 t2
where nextval('seq_progress') != 0
and t1.id = t2.id;

Once the query is running, you can open another connection to the database. To get an indication of how far it has got, you can just run the following

 select nextval('seq_progress');

Bear in mind that this will also increment it by 1 but if you have millions of rows which is really the only case in which this would be useful, a few additional increments is hardly going to make a difference.

Good luck and have fun!

Linux bulk search and replace

Doing a bulk search and replace across a set of files is actually surprisingly easy. sed is the key. It has a flag – i that will modify the files passed to it in-place.

$ sed -e 's/TextToFind/Replacement/' -i file1 file2 file3

Tie this power with either grep -l . [Thanks to Steve for pointing out a mistake in the following, now corrected]

$ grep -l TextToFind * |xargs sed -e 's/TextToFind/Replacement/' -i

or find

$ find . -exec sed -e 's/TextToFind/Replacement' -i {} ;

If there are multiple changes you want to make, just put them all into a file and pass it in via the -f flag.

file: replacements.patterns

s/TextToFind1/Replacement1/
s/TextToFind2/Replacement2/
s/TextToFind3/Replacement3/

and the command, using find to iterate through all files in the current directory and subdirectories.

find . -exec sed -f replacements.patterns -i {} ;

et voila – hope it helps.

Synergy with Linux Server & Mac Client

I  borrowed a mac to try and play with iPhone development. I already have a linux box (running Ubuntu 9.10). Anyone who has used two computers simultaneously know how annoying it is to have two keyboards/mice plugged. I originally anticipated just using X11 forwarding. However, it is an iMac with a big beautiful screen. It would be an absolute waste to not use it.

Continue reading