Monday, September 2, 2013

simulating a slow drive

I want to simulate a slow drive, to better grasp the symptoms of a database suffering from a slow disk.

Until now I used to follow up on suspicions by testing with  IOzone or sysbench/fileio , but:
  • This is sometimes difficult to do if the sysadmin is not me, and will not/can not cooperate
  • This is imperfect because we would typically do such tests at night when the SAN which I suspect is less used ;-)
I could use a SD card or something, but most of my test setups are on VM, I would prefer a software solution.
I almost started to write my own slow FUSE filesystem, when I finally found a nice trick from this post:

Use nbd, the Network Block Device, and then rate limit access to it using say trickle.
sudo apt-get install nbd-client nbd-server trickle

Knowing neither trickle nor nbd, I tested each  before doing the setup:

1) Trickle test

Trickle will limit the network speed of a command:
trickle -u <upload speed> -d <download speed> command

Example:$ wget
=> 11M/s

$ trickle -d 20 wget

=> 22 K/s

2) NBD test

NBD stands for Network Block Device. I understand it to be a simple way to use a block device from a remote machine (simpler than drbd or iscsi targets/initiators implementations)

Blocks may be files or partitions, logical volumes... Let's try to create a file :

dd if=/dev/zero of=/home/phil/apps/NBD/nbd-drive bs=1024 count=10240
Then run the nbd server on that file:
nbd-server 9500 /home/phil/apps/NBD/nbd-drive 

It seems to launch a daemon 'nbd-server', which do listen on 9500:

$ netstat -plnt
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0  *               LISTEN      2195/nbd-server 

Let's try to connect the client:

sudo nbd-client localhost 9500 /dev/nbd0

Seems to launch a daemon as well:
root      2276     1  0 07:04 ?        00:00:00 nbd-client localhost 9500 /dev/nbd0
root      2277     2  0 07:04 ?        00:00:00 [nbd0]

We can now format this device and mount it:
sudo mkfs /dev/nbd0 sudo mount /dev/nbd0 /mnt $ df -h Filesystem Size Used Avail Use% Mounted on ... /dev/nbd0 9.7M 92K 9.1M 1% /mnt

The test is OK. Let's umount and release:

sudo umount /mnt
sudo nbd-client -d /dev/nbd0
killall nbd-server

3) Trickle + NBD to make a slow drive

I succeeded by running the server through trickle this way:

$ trickle -d 20 -u 20 nbd-server 9500 /home/phil/apps/NBD/nbd-drive bs=1024 count=10240
trickle: Could not reach trickled, working independently: No such file or directory

Then started the client again with:

sudo nbd-client localhost 9500 /dev/nbd0

And indeed this block device was really slow :

$ sudo dd if=/dev/nbd0 of=/dev/null bs=65536 skip=100 count=10
10+0 records in
10+0 records out
655360 bytes (655 kB) copied, 28.5783 s, 22.9 kB/s

Finally I wrote that script to make the drive ready when I need it:

$ cat 


if [ $# -eq 0 ] ; then

  echo making/mounting slow drive
  trickle -d 20 -u 20 nbd-server 9500 $FILE_OR_PARTITION bs=1024 count=10240
  sudo nbd-client localhost 9500 /dev/nbd0
  sudo mount /dev/nbd0 $MOUNTPOINT

  if [ \( $# -eq 1 \) -a \(  $1 == "-u" \) ]; then

    echo unmountint slow drive

    sudo umount $MOUNTPOINT
    sudo nbd-client -d /dev/nbd0
    killall nbd-server

    echo  USAGE: $0 [-u]
    echo         -u : umount the slow drive

Use the -l or /etc/nbd_server.allow option of the server to restrict the clients (Otherwise any client on the network can use the block device... )

No comments:

Post a Comment