Justin's Wiki - iostat

iostat

iostat - Report

Central Processing Unit (CPU) statistics
input/output statistics for devices, partitions and network

filesystems (NFS).

The first report generated by the iostat command provides statistics concerning the time since the system was booted. Each subsequent report covers the time since the previous report.
All statistics are reported each time the iostat command is run. The report consists of a CPU header row followed by a row of CPU statistics. On multiprocessor systems, CPU statistics are calculated system-wide as averages among all processors.
A device header row is displayed followed by a line of statistics for each device that is configured. When option -n is used, an NFS header row is displayed followed by a line of statistics for each network filesystem that is mounted.

Options

-V: show version

justin@Justin:~$ iostat -V
sysstat 版本 9.0.6.1
(C) Sebastien Godard (sysstat <at> orange.fr)

no options

show both cpu and device:

justin@Justin:~$ iostat
Linux 2.6.32-5-686 (Justin)     2012年03月07日  _i686_  (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.39    0.54    9.53   43.75    0.00   40.78

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              70.09      1792.88       261.65    2957047     431545
sdb               8.51       142.10         0.03     234363         57
sdc               9.24       143.69         0.03     236987         57
sdd              12.95       240.30         0.03     396333         57
sde              14.51       512.70        17.31     845608      28544
dm-0             35.54       511.85        17.31     844202      28544

-t: show report time

show report time:

2012年03月07日 10时39分43秒

justin@Justin:~$ iostat -t
Linux 2.6.32-5-686 (Justin)     2012年03月07日  _i686_  (1 CPU)

2012年03月07日 10时39分43秒
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.44    0.53    9.50   42.43    0.00   42.10

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              67.94      1736.65       254.14    2957863     432849
sdb               8.24       137.60         0.03     234363         57
sdc               8.94       139.14         0.03     236987         57
sdd              12.54       232.70         0.03     396333         57
sde              14.18       506.36        16.76     862432      28544
dm-0             34.54       505.53        16.76     861026      28544

-k/-m

show statistics in kilo or mega bytes:

     -k   Display statistics  in kilobytes per  second instead of
          blocks per second.  Data  displayed are valid only with
          kernels 2.4 and later.

     -m   Display statistics  in megabytes per  second instead of
          blocks  or kilobytes  per second.   Data  displayed are
          valid only with kernels 2.4 and later.

justin@Justin:~$ iostat -k
Linux 2.6.32-5-686 (Justin)     2012年03月07日  _i686_  (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.50    0.47    9.21   38.04    0.00   46.77

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              60.97       778.57       114.25    1478939     217016
sdb               7.39        61.69         0.02     117181         28
sdc               8.02        62.38         0.02     118493         28
sdd              11.24       104.32         0.02     198166         28
sde              12.72       227.01         7.51     431216      14272
dm-0             30.97       226.64         7.51     430513      14272

justin@Justin:~$ iostat
Linux 2.6.32-5-686 (Justin)     2012年03月07日  _i686_  (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.48    0.47    9.19   37.94    0.00   46.91

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              60.79      1552.38       227.86    2957879     434169
sdb               7.37       123.00         0.03     234363         57
sdc               8.00       124.38         0.03     236987         57
sdd              11.21       208.01         0.03     396333         57
sde              12.68       452.63        14.98     862432      28544
dm-0             30.88       451.89        14.98     861026      28544

1 Blk = 512B ! phyiscal block

-c/-d: show cpu/device statistics

[interval [count]]

repeat

-z: omit no activity

z Tell iostat to omit output for any devices for which
there was no activity during the sample period.

-x: extented statistics

x Display extended statistics. This option works with
post 2.5 kernels since it needs /proc/diskstats file or a mounted sysfs to get the statistics. This option may also work with older kernels (e.g. 2.4) only if extended statistics are available in /proc/partitions (the kernel needs to be patched for that).

-n: display nfs report

n Display the network filesystem (NFS) report. This

option works only with kernel 2.6.17 and later.
h Make the NFS report displayed by option -n easier to

read by a human.

justin@Justin:~$ iostat -n -h
Linux 2.6.32-5-686 (Justin)     2012年03月07日  _i686_  (1 CPU)

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s   wBlk_dir/s   rBlk_svr/s   wBlk_svr/s     ops/s    rops/s    wops/s

justin@Justin:~$ iostat -n
Linux 2.6.32-5-686 (Justin)     2012年03月07日  _i686_  (1 CPU)

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s   wBlk_dir/s   rBlk_svr/s   wBlk_svr/s     ops/s    rops/s    wops/s

seems no difference?

-N: display the registered device mapper name

N Display the registered device mapper names for any

device mapper devices. Useful for viewing LVM2 statistics.

-p show all device and sub partitions

p [ { device [,...] | ALL } ]

The -p option displays statistics for block devices and all their partitions that are used by the system. If a device name is entered on the command line, then statistics for it and all its partitions are displayed. Last, the ALL keyword indicates that statistics have to be displayed for all the block devices and partitions defined by the system, including those that have never been used. Note that this option works only with post 2.5 kernels.

justin@Justin:~$ iostat -p /dev/sda,/dev/sdb
Linux 2.6.32-5-686 (Justin)     2012年03月07日  _i686_  (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.39    0.31    7.97   25.00    0.00   61.34

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              40.21      1020.57       153.10    2958039     443745
sda1              6.15       204.20       144.70     591860     419400
sda2              0.05         0.44         0.00       1270          0
sda3             34.00       815.78         8.40    2364477      24345
sdb               4.84        80.86         0.02     234363         57
sdb1              4.83        80.65         0.02     233771         57

partition or whole disk device

all the above command can append partition or disk device file:

justin@Justin:~$ iostat /dev/sda /dev/sdb1
Linux 2.6.32-5-686 (Justin)     2012年03月07日  _i686_  (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.45    0.30    7.98   23.99    0.00   62.28

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              38.62       978.89       147.58    2958159     445985
sdb1              4.64        77.36         0.02     233771         57

Displayed Info

CPU

justin@Justin:~$ iostat -c
Linux 2.6.32-5-686 (Justin)     2012年03月07日  _i686_  (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.08    0.67    9.21   46.17    0.00   38.86

CPU Utilization Report The first report generated by the iostat command is the CPU Utilization Report. For multiprocessor systems, the CPU values are global averages among all processors. The report has the following format:

%user Show the percentage of CPU utilization that occurred while executing at the user level (application).

%nice Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.

%system Show the percentage of CPU utilization that occurred while executing at the system level (kernel).

%iowait Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.

%steal Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor. (虚拟机相关）

%idle Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.

Disk

Device Utilization Report The second report generated by the iostat command is the Device Utilization Report. The device report provides statistics on a per physical device or partition basis. Block devices and partitions for which statistics are to be displayed may be entered on the command line. If no device nor partition is entered, then statistics are displayed for every device used by the system, and providing that the kernel maintains statistics for it. If the ALL keyword is given on the command line, then statistics are displayed for every device defined by the system, including those that have never been used. The report may show the following fields, depending on the flags used:

Device: This column gives the device (or partition) name, which is displayed as hdiskn with 2.2 kernels, for the nth device. It is displayed as devm-n with 2.4 kernels, where m is the major number of the device, and n a distinctive number. With newer kernels, the device name as listed in the /dev directory is displayed.

tps Indicate the number of transfers per second that were issued to the device. A transfer is an I/O request to the device. Multiple logical requests can be combined into a single I/O request to the device. A transfer is of indeterminate size.

Blk_read/s Indicate the amount of data read from the device expressed in a number of blocks per second. Blocks are equivalent to sectors with kernels 2.4 and later and therefore have a size of 512 bytes. With older kernels, a block is of indeterminate size.

Blk_wrtn/s Indicate the amount of data written to the device expressed in a number of blocks per second.

Blk_read The total number of blocks read.

Blk_wrtn The total number of blocks written.

kB_read/s Indicate the amount of data read from the device expressed in kilobytes per second.

kB_wrtn/s Indicate the amount of data written to the device expressed in kilobytes per second.

kB_read The total number of kilobytes read.

kB_wrtn The total number of kilobytes written.

MB_read/s Indicate the amount of data read from the device expressed in megabytes per second.

MB_wrtn/s Indicate the amount of data written to the device expressed in megabytes per second.

MB_read The total number of megabytes read.

MB_wrtn The total number of megabytes written.

长期超大数据读写不正常，会影响系统性能。

Extended Disk

rrqm/s The number of read requests merged per second that were queued to the device.

wrqm/s The number of write requests merged per second that were queued to the device.

r/s The number of read requests that were issued to the device per second.

w/s The number of write requests that were issued to the device per second.

rsec/s The number of sectors read from the device per second.

wsec/s The number of sectors written to the device per second.


          rkB/s
               The number  of kilobytes read from  the device per
               second.

          wkB/s
               The number of kilobytes  written to the device per
               second.

          rMB/s
               The number  of megabytes read from  the device per
               second.

          wMB/s
               The number of megabytes  written to the device per
               second.

1sec = 512b

justin@Justin:~$ iostat -x -k
Linux 2.6.32-5-686 (Justin)     2012年03月07日  _i686_  (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.85    0.22    8.57   18.76    0.00   65.60

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda             183.25    12.39   27.06    2.41   374.09    59.15    29.41     0.67   22.89   6.34  18.67
sdb               3.92     0.00    3.42    0.00    28.55     0.01    16.70     0.01    2.14   1.54   0.53
sdc               3.71     0.00    3.71    0.00    28.86     0.01    15.56     0.01    2.77   1.99   0.74
sdd               7.02     0.00    5.20    0.00    48.27     0.01    18.56     0.02    3.18   2.13   1.11
sde               8.05     0.58    5.97    0.36   118.65     3.58    38.63     0.46   72.09  13.48   8.53
dm-0              0.00     0.00   13.92    0.90   118.48     3.58    16.48     1.02   69.18   5.73   8.49

justin@Justin:~$ iostat -x
Linux 2.6.32-5-686 (Justin)     2012年03月07日  _i686_  (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.85    0.22    8.57   18.76    0.00   65.61

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda             183.16    12.39   27.05    2.40   747.85   118.25    29.41     0.67   22.89   6.34  18.66
sdb               3.91     0.00    3.42    0.00    57.07     0.01    16.70     0.01    2.14   1.54   0.53
sdc               3.70     0.00    3.71    0.00    57.70     0.01    15.56     0.01    2.77   1.99   0.74
sdd               7.02     0.00    5.20    0.00    96.50     0.01    18.56     0.02    3.18   2.13   1.11
sde               8.04     0.58    5.96    0.36   237.19     7.16    38.63     0.46   72.09  13.48   8.52
dm-0              0.00     0.00   13.91    0.90   236.85     7.16    16.48     1.02   69.18   5.73   8.49

avgrq-sz The average size (in sectors) of the requests that were issued to the device.

avgqu-sz The average queue length of the requests that were issued to the device.

await The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.

svctm The average service time (in milliseconds) for I/O requests that were issued to the device.

%util Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.

justin@Justin:~$ iostat -x && iostat
Linux 2.6.32-5-686 (Justin)     2012年03月07日  _i686_  (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.94    0.21    8.63   18.17    0.00   66.04

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda             177.35    12.02   26.19    2.36   724.13   114.96    29.39     0.65   22.86   6.33  18.07
sdb               3.79     0.00    3.31    0.00    55.26     0.01    16.70     0.01    2.14   1.54   0.51
sdc               3.59     0.00    3.59    0.00    55.87     0.01    15.56     0.01    2.77   1.99   0.72
sdd               6.79     0.00    5.03    0.00    93.44     0.01    18.56     0.02    3.18   2.13   1.07
sde               7.79     0.56    5.80    0.35   231.11     6.94    38.70     0.44   71.95  13.45   8.27
dm-0              0.00     0.00   13.50    0.87   230.78     6.94    16.54     0.99   69.11   5.74   8.24

Linux 2.6.32-5-686 (Justin)     2012年03月07日  _i686_  (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.94    0.21    8.63   18.17    0.00   66.04

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              28.55       724.12       114.96    3071383     487609
sdb               3.31        55.26         0.01     234379         57
sdc               3.59        55.87         0.01     236987         57
sdd               5.03        93.44         0.01     396333         57
sde               6.15       231.11         6.94     980240      29416
dm-0             14.37       230.77         6.94     978834      29416

Metrics

tps=r/s+w/s

Blk_read/s = rsec/s Blk_wrtn/s = wsec/s

iostat 结果解析

[root@20081006-1724 ~]# iostat -x Linux 2.6.9-78.ELsmp (20081006-1724) 11/20/2009

avg-cpu: %user %nice %sys %iowait %idle

19 0.00 0.04 0.03 99.73

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.05 17.60 1.46 7.72 80.69 202.57 40.34 101.29 30.87 0.01 1.06 0.37 0.34 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 29.90 0.00 3.14 3.14 0.00 sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 16.25 0.00 1.51 1.30 0.00 sda3 0.05 17.60 1.46 7.72 80.69 202.57 40.34 101.29 30.87 0.01 1.06 0.37 0.34 dm-0 0.00 0.00 1.46 25.28 80.32 202.26 40.16 101.13 10.57 0.36 13.56 0.13 0.34 dm-1 0.00 0.00 0.05 0.04 0.37 0.32 0.18 0.16 8.00 0.00 6.84 1.30 0.01

rrqm/s: 每秒进行 merge 的读操作数目。即 delta(rmerge)/s wrqm/s: 每秒进行 merge 的写操作数目。即 delta(wmerge)/s

r/s: 每秒完成的读 I/O 设备次数。即 delta(rio)/s w/s: 每秒完成的写 I/O 设备次数。即 delta(wio)/s

rsec/s: 每秒读扇区数。即 delta(rsect)/s wsec/s: 每秒写扇区数。即 delta(wsect)/s

rkB/s: 每秒读K字节数。是 rsect/s 的一半， 因为每扇区大小为512字节。 wkB/s: 每秒写K字节数。是 wsect/s 的一半。

avgrq-sz: 平均每次设备I/O操作的数据大小 (扇区)。即 delta(rsect+wsect)/delta(rio+wio)

avgqu-sz: 平均I/O队列长度。即 delta(aveq)/s/1000 (因为aveq的单位为毫秒)。

await: 平均每次设备I/O操作的等待时间 (毫秒)。即 delta(ruse+wuse)/delta(rio+wio) svctm: 平均每次设备I/O操作的服务时间 (毫秒)。即 delta(use)/delta(rio+wio)

%util: 一秒中有百分之多少的时间用于 I/O 操作，或者说一秒中有多少时间 I/O 队列是非空的。即 delta(use)/s/1000 (因为use的单位为毫秒)

justin@Justin:~$ iostat -x && iostat
Linux 2.6.32-5-686 (Justin)     2012年03月07日  _i686_  (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.94    0.21    8.63   18.17    0.00   66.04

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda             177.35    12.02   26.19    2.36   724.13   114.96    29.39     0.65   22.86   6.33  18.07
sdb               3.79     0.00    3.31    0.00    55.26     0.01    16.70     0.01    2.14   1.54   0.51
sdc               3.59     0.00    3.59    0.00    55.87     0.01    15.56     0.01    2.77   1.99   0.72
sdd               6.79     0.00    5.03    0.00    93.44     0.01    18.56     0.02    3.18   2.13   1.07
sde               7.79     0.56    5.80    0.35   231.11     6.94    38.70     0.44   71.95  13.45   8.27
dm-0              0.00     0.00   13.50    0.87   230.78     6.94    16.54     0.99   69.11   5.74   8.24

Linux 2.6.32-5-686 (Justin)     2012年03月07日  _i686_  (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.94    0.21    8.63   18.17    0.00   66.04

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              28.55       724.12       114.96    3071383     487609
sdb               3.31        55.26         0.01     234379         57
sdc               3.59        55.87         0.01     236987         57
sdd               5.03        93.44         0.01     396333         57
sde               6.15       231.11         6.94     980240      29416
dm-0             14.37       230.77         6.94     978834      29416

比较重要的参数 %util: 一秒中有百分之多少的时间用于 I/O 操作，或者说一秒中有多少时间 I/O 队列是非空的 svctm: 平均每次设备I/O操作的服务时间 await: 平均每次设备I/O操作的等待时间 avgqu-sz: 平均I/O队列长度

如果%util接近100%,表明i/o请求太多,i/o系统已经满负荷,磁盘可能存在瓶颈,一般%util大于70%,i/o压力就比较大,读取速度有较多的wait.同时可以结合vmstat查看查看b参数(等待资源的进程数)和 wa参数(IO等待所占用的CPU时间的百分比,高过30%时IO压力高)。

形象的比喻 r/s+w/s 类似于交款人的总数平均队列长度(avgqu-sz)类似于单位时间里平均排队人的个数平均服务时间(svctm)类似于收银员的收款速度平均等待时间(await)类似于平均每人的等待时间平均I/O数据(avgrq-sz)类似于平均每人所买的东西多少 I/O 操作率 (%util)类似于收款台前有人排队的时间比例

设备IO操作:总IO(io)/s = r/s(读) +w/s(写) =1.46 + 25.28=26.74

平均每次设备I/O操作只需要0.36毫秒完成,现在却需要10.57毫秒完成，因为发出的请求太多(每秒26.74个)，假如请求时同时发出的，可以这样计算平均等待时间: 平均等待时间=单个I/O服务器时间*(1+2+...+请求总数-1)/请求总数每秒发出的I/0请求很多,但是平均队列就4,表示这些请求比较均匀,大部分处理还是比较及时

await > svctm svctm 一般要小于 await (因为同时等待的请求的等待时间被重复计算了)，

svctm 的大小一般和磁盘性能有关，CPU/内存的负荷也会对其有影响，请求过多也会间接导致 svctm 的增加。

await 的大小一般取决于: 1. 服务时间(svctm) 2. I/O 队列的长度 3. I/O 请求的发出模式。如果 svctm 比较接近 await，说明 I/O 几乎没有等待时间；如果 await 远大于 svctm，说明 I/O 队列太长，应用得到的响应时间变慢。

如果 svctm 比较接近 await，说明I/O 几乎没有等待时间；如果 await 远大于 svctm，说明 I/O 队列太长，应用得到的响应时间变慢，如果响应时间超过了用户可以容许的范围，这时可以考虑更换更快的磁盘，调整内核 elevator 算法，优化应用，或者升级 CPU。

队列长度(avgqu-sz)也可作为衡量系统 I/O 负荷的指标，但由于 avgqu-sz 是按照单位时间的平均值，所以不能反映瞬间的 I/O 洪水。

NFS

Network Filesystem report The Network Filesystem (NFS) report provides statistics for each mounted network filesystem. The report shows the following fields:

Filesystem: This columns shows the hostname of the NFS server followed by a colon and by the directory name where the network filesystem is mounted.

rBlk_nor/s Indicate the number of blocks read by applications via the read(2) system call interface. A block has a size of 512 bytes.

wBlk_nor/s Indicate the number of blocks written by applications via the write(2) system call interface.

rBlk_dir/s Indicate the number of blocks read from files opened with the O_DIRECT flag.

wBlk_dir/s Indicate the number of blocks written to files opened with the O_DIRECT flag.

rBlk_svr/s Indicate the number of blocks read from the server by the NFS client via an NFS READ request.

wBlk_svr/s Indicate the number of blocks written to the server by the NFS client via an NFS WRITE request.

rkB_nor/s Indicate the number of kilobytes read by applications via the read(2) system call interface.

wkB_nor/s Indicate the number of kilobytes written by applications via the write(2) system call interface.

rkB_dir/s Indicate the number of kilobytes read from files opened with the O_DIRECT flag.

wkB_dir/s Indicate the number of kilobytes written to files opened with the O_DIRECT flag.

rkB_svr/s Indicate the number of kilobytes read from the server by the NFS client via an NFS READ request.

wkB_svr/s Indicate the number of kilobytes written to the server by the NFS client via an NFS WRITE request.

rMB_nor/s Indicate the number of megabytes read by applications via the read(2) system call interface.

wMB_nor/s Indicate the number of megabytes written by applications via the write(2) system call interface.

rMB_dir/s Indicate the number of megabytes read from files opened with the O_DIRECT flag.

wMB_dir/s Indicate the number of megabytes written to files opened with the O_DIRECT flag.

rMB_svr/s Indicate the number of megabytes read from the server by the NFS client via an NFS READ request.

wMB_svr/s Indicate the number of megabytes written to the server by the NFS client via an NFS WRITE request.

ops/s Indicate the number of operations that were issued to the filesystem per second.

rops/s Indicate the number of 'read' operations that were issued to the filesystem per second.

wops/s Indicate the number of 'write' operations that were issued to the filesystem per second.