AIX performance tuning

jpzhai@gmail.com · *nix forums beginner Joined: 21 Apr 2006 Posts: 3

vmstat:
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
0 1 88492 151331 0 0 0 0 0 0 13 663 84 2 1 0 96
0 1 88496 151327 0 0 0 0 0 0 7 411 68 2 1 0 97
1 1 88496 151327 0 0 0 0 0 0 7 651 73 2 2 0 96
1 1 88496 151327 0 0 0 0 0 0 9 412 71 2 1 0 97
2 1 88500 151323 0 0 0 0 0 0 12 652 77 4 1 0 95
0 1 88500 151323 0 0 0 0 0 0 7 405 72 1 2 0 98
1 1 88501 151322 0 0 0 0 0 0 8 657 69 2 2 0 97
1 1 88502 151321 0 0 0 0 0 0 7 416 69 0 0 0 99
1 1 88502 151321 0 0 0 0 0 0 9 628 75 1 1 0 99
2 1 88504 151319 0 0 0 0 0 0 12 452 84 1 0 0 99
iostat:
tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 394.0 0.0 1.0 0.0 99.0

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.0 0.0 0.0 0 0
hdisk1 0.0 0.0 0.0 0 0
cd0 0.0 0.0 0.0 0 0
uptime:
09:25AM up 26 mins, 1 user, load average: 0.17, 0.19, 0.18
Can anyone pls tell me where is the bottleneck?
I dont know why the wait% of the CPU is so high.
ths

Hajo Ehlers · *nix forums Guru Joined: 23 Sep 2005 Posts: 344

jpzhai@gmail.com wrote:

praveen · *nix forums beginner Joined: 27 Dec 2005 Posts: 26

Hajo Ehlers wrote:

steve.stamper@foremost.co · *nix forums beginner Joined: 30 Jun 2006 Posts: 2

jpzhai@gmail.com wrote:

Mark Taylor · *nix forums Guru Wannabe Joined: 21 Feb 2005 Posts: 292

Don't get caught up in this whole wait i/o thing. a single cpu system
with 1 i/o outstanding and no other runable threads (i.e. idle) will
have 100% wait i/o. There was a big discussion a couple of years ago on
removing the kernel tick as it has confused many many many techs.

So, if you have only 1 cpu, then you are going to have high wait i.o
figures, it does not mean your disk subsystem is slow.

First thing you need to do is work out where all the i/o when you hit
high wait i/o times, you can.

1. use iostat or nmon to get your disk busy % then look at what is on
that disk.
2. use filemon to get detaillled stats about the file / lv thats most
heavily used when you see the wait i/o shoot up.

Like someone has said above, identify the problem file / lv and spread
that across as many spindles / adapters as possible to get some
parallelism going.

Also, make sure you have enough aio servers defined ( or posix_aio
depending if your app uses them )

It may be work benchmarking your subsystem so that you know what it is
capable of, this may be a simple dd if=/dev/zero of=test.img bs=1024
count=10240000 & and check out nmon (a == adapter sub command) or it
could be a complex as buying some commercial software ..

Nigle has kindly put up some of his home grown software that can stress
disks as part of the nstress kit

Ref:
http://www-941.ibm.com/collaboration/wiki/display/WikiPtype/nstress

HTH
Mark Taylor

Mark Taylor · *nix forums Guru Wannabe Joined: 21 Feb 2005 Posts: 292

Actually, looking at the iostat output your disks are not busy at all
(if thats all the disks on the system and your post was not truncated
that is) ..

check the errpt, see any disk errors ?

If not, then you may want to run a kernel trace and egrep out "return
from kwrite|return from kread" to see which procs are doing the i/o ..
this should point you in the right direction etc..

Rgds
MarkT

Google