How to fight a fork bomb ?

Last evening I came across this very elegant fork bomb:

:(){ :|:& };:

I thought I'd try it out ;-)

As it turns out, on my RHEL 5.6 system it does mess up, but rapidly dies out:

[phil@mylinux ~]$ :(){ :|: &}; :
-bash: fork: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable
[ repeated many many times ]
[1]+  Done                    : | :


However this modified version does eat my resources for a while!

[phil@mylinux ~]$ :(){ sleep 1 ; :|: &}; :

In a root window I observ the damages:
[root@mylinux ~]# ps -eaf | wc -l
331
[root@mylinux ~]# ps -eaf | wc -l
811
[root@mylinux ~]# ps -eaf | wc -l
2662
[root@mylinux ~]# ps -eaf | wc -l
9072
[root@mylinux ~]# ps -eaf | wc -l
32493

then sometimes I would get:
[root@mylinux ~]# ps -eaf | wc -l
-bash: fork: Resource temporarily unavailable

I thought for a while how to fight this the best and quickest way on a production system

Obvisouly it's cumbersome to type quickly something like:
for I in $(ps -fu phil | awk  ' { print $2} ' ) ; do kill -9 $I; done
and it uses several processes when they're scarce.

then I remembered about 'killall'

killall -u phil       # should send a INT signal to all processes

killall -u phil -s 9  # maybe better in this situation, send the KILL signal !


I tried the command, but some processes are still there !
I think that during the execution of 'killall' some more processes are generated...

In the end I was able to stop all this simply by repeating the command quickly

[root@mylinux ~]# killall -u phil -s 9
[root@mylinux ~]# killall -u phil -s 9
[root@mylinux ~]# killall -u phil -s 9
[root@mylinux ~]# killall -u phil -s 9
[root@mylinux ~]# killall -u phil -s 9
[root@mylinux ~]# killall -u phil -s 9
[root@mylinux ~]# killall -u phil -s 9
[root@mylinux ~]# killall -u phil -s 9
[root@mylinux ~]# ps -eaf | wc -l
296

Good to know. I will be less nervous if it happens on a productive system!

block size matters for disk... and memory

Having a too large block waste disk space (because only part of the block is filled), but also memory cache.
I verified that through a few simple tests.

On this Linux ext3 filesystem I have 4K blocks:

[root@mylinux test]# tune2fs -l /dev/VolGroup00/LogVol00
tune2fs 1.39 (29-May-2006)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          89e74fff-184b-489e-b83b-981a31b66756
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              76840960
Block count:              76816384
Reserved block count:     3840819
Free blocks:              61196276
Free inodes:              75955279
First block:              0
Block size:               4096
Fragment size:            4096
...

• Let's create 100000 files of 1 byte, and observ the space used:

[root@mylinux test]# df /; for i in {1..100000} ; do echo  > ./file$i ; done ; df /


Filesystem                          1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00     297640376  59261952 223015148  21% /
Filesystem                          1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00     297640376  59665048 222612052  22% /


[root@mylinux test]# echo $((59665048-59261952))
403096


=> We're using ~ 400 MB on the disk (for 0.1MB of data)

Interestingly, 'ls' will show the proper file size, but disk usage 'du' reports the amount really used on disk:

[root@mylinux test]# ls -l | head
total 800000
-rw-r--r-- 1 root root 1 Jan 25 13:52 file1
-rw-r--r-- 1 root root 1 Jan 25 13:52 file10
-rw-r--r-- 1 root root 1 Jan 25 13:52 file100
-rw-r--r-- 1 root root 1 Jan 25 13:52 file1000
-rw-r--r-- 1 root root 1 Jan 25 13:52 file10000
-rw-r--r-- 1 root root 1 Jan 25 13:53 file100000
-rw-r--r-- 1 root root 1 Jan 25 13:52 file10001
-rw-r--r-- 1 root root 1 Jan 25 13:52 file10002
-rw-r--r-- 1 root root 1 Jan 25 13:52 file10003
[root@mylinux test]# ls | wc -l
100000
[root@mylinux test]# du -sh .
784M    .

That is what we want, but it's an example where the sum of 'ls' sizes don't match 'du' or 'df' usage

• But perhaps more annoying (and less known) is the cache use:

Let's clear the cache, as much as we can:

[root@mylinux test]# sync ; echo 1 > /proc/sys/vm/drop_caches

And read all these files

[root@mylinux test]# free ; for i in {1..100000} ; do cat ./file$i > /dev/null ; done ; free


             total       used       free     shared    buffers     cached
Mem:      32935792     573268   32362524          0        424      22632
-/+ buffers/cache:     550212   32385580
Swap:     34996216          0   34996216


             total       used       free     shared    buffers     cached
Mem:      32935792    1042208   31893584          0      15640     471984
-/+ buffers/cache:     554584   32381208
Swap:     34996216          0   34996216


[root@mylinux test]# echo $((471984 - 22632))
449352

=>We're also using ~400MB of memory cache for 0.1MB of data!
That is because the cache contain whole blocks. So a large block could also cause cache waste.

(The number is larger because OS activities also used some cache during the test, including reading directory etc.)

• Now if the files are closer to the block size: Let's create 100000 files of  about 3KB :

(Note I'm not using 4KB because a file of exactly 4K will need 2 blocks because of the i-node structure)

[root@mylinux test]# df /; for i in {1..100000} ; do dd if=/dev/sda bs=3000 count=1 of=./medfile$i &> /dev/null ; done; df /


Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     297640376  59665088 222612012  22% /
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     297640376  60068324 222208776  22% /
[root@mylinux test]# echo $((60068324-59665088))
403236


[root@mylinux test]# sync ; echo 1 > /proc/sys/vm/drop_caches


[root@mylinux test]# free ;for i in {1..100000} ; do cat ./medfile$i > /dev/null ; done ; free


             total       used       free     shared    buffers     cached
Mem:      32935792     867868   32067924          0        420      22468
-/+ buffers/cache:     844980   32090812
Swap:     34996216          0   34996216
             total       used       free     shared    buffers     cached
Mem:      32935792    1282484   31653308          0      14696     423976
-/+ buffers/cache:     843812   32091980
Swap:     34996216          0   34996216


[root@mylinux test]# echo $((423976-22468))
401508


Again we used 400MB of disk space, and cache. But this time to address ~ 300MB of data


This might be of  some importance in some cases.

I am  presently trying to optimize a very large source tree compilation (200 000 files), and with a more adjusted block size it helps to fit everything in the memory cache and hopefully improve performances.

block size / files size

Today I was wondering about the optimal blocksize and typical file sizes

The filesystem I am presenlty concerned about contains our big source tree (Java), and its compiled classes. I am trying to optimize the continuous compilation and integration system (Jenkins). This things report to developpers wether the build is broken or tests fail. We want it to run as frequently as possible.

I wrote a short python script to find file size ranges:


#!/usr/bin/python
import os
d=os.popen("find . -type f -exec ls -ld {} \; ")      # ls all files through find


sizes= [ int( line.split(' ')[4] ) for line in d.readlines() ]      #size is 4th field in ls output: '-rwxr-xr-x 1 hudson hudson 120 Jan 23 15:56 ./file1'
d.close()


keys=[ pow(2,n) for n in range(8,32)]      # [ 256,512,1024....
NbFilesPerSizeRange={}    # this will contain count for each range, starting with zero
for k in keys :
NbFilesPerSizeRange[k]=0


for s in sizes:                          
for srange in keys:       #then just count
if s< srange :
NbFilesPerSizeRange[srange]+=1
break


csvTxt="".join([ "%d\n" %NbFilesPerSizeRange[s] for s in keys ])
print csvTxt

file sizes in Java + classes tree of our project

It turns out the default suggested by mkfs  ( 4096 bytes block for that disk) is not too bad.

While I was at it, I wondered what are the typical file sizes in the Linux distribution?

( Here it is Red Hat RHEL5.6 )

I slightly modified the script and got this result:

                          

file sizes in a Red Hat distribution