Results Interpretation
Raw Results
Details
We compare results from five kernels, run on the same hardware platform.
AIM9 reports results using three metrics:
AIM9 reports these metrics for the 60 routines performed by the test
suite.
The tests divide into several groups.
- Arithmetic Addition, divison and multiplication of short, long, int,float and double values. Results are reported as thousands of operations per second.
- Disk/Filesystem IO Does some testing of IO performance, primarily tests the low-level system calls, such as create/close link/unlink. File sizes are small, results here are not especially comparable to other IO tests
- InterProcess Communication Passing small messages through TCP, UDP, FIFO's, shared memory and several different PIPEs and a range of data sizes. These tests show noticeable change between kernel versions
- Library/System A miscellanous collection of tests. Includes some trig functions, and simple shell script tests. Also tests for setjmp() and longjmp()
- Memory and Process Mangement Tests these calls: brk() exec() fork() plus a memory paging test. Fork performance varies widely by kernel
- Function Calls Tests a function with increasing number of function arguments. Appears to be insensative to kernel changes
We compare tests run on different kernels, using one filesystem type (ext2)
Tests run on one kernel can be used to compare filesystem performance.
For each metric, we calculate the average score and
the standard deviation for the five kernels. Then to scale the results,
we calculate the ratio between the standard deviation and the average,
expressed as percent ( (StdDev/Average) * 100 ) This comparison revels
which test are sensitive to kernel changes. In the following charts,
we have color coded the results and grouped them into three areas:
- Greater than 10% change
- Between 5% and 10%
- Between 1% and 5%
Comments on the operations chart
Change greater than 10 percent
- These tests appear to be quite sensitive to kernel changes
- Test 7 page_test Uses sbrk(), allocates, deallocates and
dirties process memory space. Grabs 1 megabyte of memory and moves
through it in 4k chunks.
- Test 8 brk_test Uses sbrk() allocates and deallocates
process memory but does not dirty it. Also uses 4k sized chunks.
- Test 11 exec_test Forks a child, does execl() of "./true"
- parent does a wait() and recovers child exit status. Notice the
dramatic change between kernel 2.4.9 and 2.4.18 -
- Test 12 fork_test Forks a child, child calls exit(0),
parent does a wait(), gets child exit status. Notice the improvement
in kernel 2.4.20
- Tests 49-51 Shell tests These test do very, very little
( ls -l > /dev/null 2>&1) Notice the delta between 2.4.9 and 2.4.18.
Why?
Change between 5 and 10 percent
- Tests 54-59 use a common routine to actually move data. The routine
uses data packets across a range of sizes.
- Test 55 udp_test Creates two datagram sockets ( socket(AF_INET,
SOCK_DGRAM, 0) ) and sends data between them. None of the new kernels
match 2.4.9 performance
- Test 56 fifo_test Creates a FIFO, opens a read descriptor
and a write descriptor and sends data. Again, 2.4.9 outperforms
recent kernels.
- Test 57 stream_pipe Creates a stream socket pair, sends
data. ( socketpair(AF_UNIX, SOCK_STREAM, 0, pipefd) )
Test 59 pipe_cpy Uses the pipe() call to create a pipe,
transfers data.
Change between 1 and 5 percent
- Test 6 creat-close uses the creat() and close() calls
in a loop, 1000 times. (creat() in Linux should call open() w/O_CREAT
)
- Test 10 signal_test Installs a handler for SIGUSR2, send
the signal multiple times in a loop
- Test 13 link_test Creates a file, add 7 links to the
file, remove and re-attach the links. May require directory synch
Tests 14-18 disk_* Small disk tests, sequential/random
reads/writes
Tests 20,21 sync_disk_* Read and write tests with O_SYNC
- Test 22 disk_src Directory searches
- Test 44 mem_rtns_1 Tests of malloc and calloc functions
- Test 47 misc_rtns_1 Tests getpid(), getpgrp() and related
functions
- Test 53 shared_memory tests mmap, flock and semaphore
operations
- Test 54 tcp_test socket communications test using - socket(AF_INET,
SOCK_STREAM, 0);
- Test 58 dgram_pipe Datagram socket communication test
- socketpair(AF_UNIX, SOCK_DGRAM, 0, pipefd)
Change less than 1 percent
- Tests 1-5 These tests do basic math operations, namely
addition. We hope that kernel changes never affect these tests.
- Test 9 Test of setjmp() and longjmp() AIM believes these
calls are used extensively in threads implementation - might be
one to watch
- Test 19 Sync sequential disk writes. (file created with
O_SYNC ) this might be another test to watch for future changes
- Tests 23-27 More basic math, testing division
- Tests 28-31 These tests call a function with a varying
number of parameters. Again, the kernel shouldn't impact this
- Test 32 Finding prime numbers, more integer math
- Tests 33-37 Multiplication
- Tests 38-42 A long list of math functions, matrix and
array functions Again, not very kernel-dependant
- Tests 43,45-46 These tests are interesting for their
lack of variation. They test basic memory and string functions,
(memset,memcpy, memmove,strcat,strcpy) However these tests use very
small sizes ( largest alloc is 8192 bytes) and this may explain
the lack of sensativity to kernel changes.
- Test 48 Opens a directory ("/bin") , reads and rewinds
it multiple times. Again, interesting for the lack of variety.
- Test 52 logrhythm functions
- Test 60 Ram copy. Tests memcpy, but again the largest
size used is 8192 bytes