高负载

TODO 优化

节点高负载会导致进程无法获得足够的 cpu 时间片来运行，通常表现为网络 timeout，健康检查失败，服务不可用。

过多 IO 等待

有时候即便 cpu ‘us’ (user) 不高但 cpu ‘id’ (idle) 很高的情况节点负载也很高，这是为什么呢？通常是文件 IO 性能达到瓶颈导致 IO WAIT 过多，从而使得节点整体负载升高，影响其它进程的性能。

使用 top 命令看下当前负载：

  1. top - 19:42:06 up 23:59,  2 users,  load average: 34.64, 35.80, 35.76
  2. Tasks: 679 total,   1 running, 678 sleeping,   0 stopped,   0 zombie
  3. Cpu(s): 15.6%us,  1.7%sy,  0.0%ni, 74.7%id,  7.9%wa,  0.0%hi,  0.1%si,  0.0%st
  4. Mem:  32865032k total, 30989168k used,  1875864k free,   370748k buffers
  5. Swap:  8388604k total,     5440k used,  8383164k free,  7982424k cached
  7.   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  8.  9783 mysql     20   0 17.3g  16g 8104 S 186.9 52.3   3752:33 mysqld
  9.  5700 nginx     20   0 1330m  66m 9496 S  8.9  0.2   0:20.82 php-fpm
  10.  6424 nginx     20   0 1330m  65m 8372 S  8.3  0.2   0:04.97 php-fpm
  11.  6573 nginx     20   0 1330m  64m 7368 S  8.3  0.2   0:01.49 php-fpm
  12.  5927 nginx     20   0 1320m  56m 9272 S  7.6  0.2   0:12.54 php-fpm
  13.  5956 nginx     20   0 1330m  65m 8500 S  7.6  0.2   0:12.70 php-fpm
  14.  6126 nginx     20   0 1321m  57m 8964 S  7.3  0.2   0:09.72 php-fpm
  15.  6127 nginx     20   0 1319m  54m 9520 S  6.6  0.2   0:08.73 php-fpm
  16.  6131 nginx     20   0 1320m  56m 9404 S  6.6  0.2   0:09.43 php-fpm
  17.  6174 nginx     20   0 1321m  56m 8444 S  6.3  0.2   0:08.92 php-fpm
  18.  5790 nginx     20   0 1319m  54m 9468 S  5.6  0.2   0:17.33 php-fpm
  19.  6575 nginx     20   0 1320m  55m 8212 S  5.6  0.2   0:02.11 php-fpm
  20.  6160 nginx     20   0 1310m  44m 8296 S  4.0  0.1   0:10.05 php-fpm
  21.  5597 nginx     20   0 1310m  46m 9556 S  3.6  0.1   0:21.03 php-fpm
  22.  5786 nginx     20   0 1310m  45m 8528 S  3.6  0.1   0:15.53 php-fpm
  23.  5797 nginx     20   0 1310m  46m 9444 S  3.6  0.1   0:14.02 php-fpm
  24.  6158 nginx     20   0 1310m  45m 8324 S  3.6  0.1   0:10.20 php-fpm
  25.  5698 nginx     20   0 1310m  46m 9184 S  3.3  0.1   0:20.62 php-fpm
  26.  5779 nginx     20   0 1309m  44m 8336 S  3.3  0.1   0:15.34 php-fpm
  27.  6540 nginx     20   0 1306m  40m 7884 S  3.3  0.1   0:02.46 php-fpm
  28.  5553 nginx     20   0 1300m  36m 9568 S  3.0  0.1   0:21.58 php-fpm
  29.  5722 nginx     20   0 1310m  45m 8552 S  3.0  0.1   0:17.25 php-fpm
  30.  5920 nginx     20   0 1302m  36m 8208 S  3.0  0.1   0:14.23 php-fpm
  31.  6432 nginx     20   0 1310m  45m 8420 S  3.0  0.1   0:05.86 php-fpm
  32.  5285 nginx     20   0 1302m  38m 9696 S  2.7  0.1   0:23.41 php-fpm

wa (wait) 表示 IO WAIT 的 cpu 占用，默认看到的是所有核的平均值，要看每个核的 wa 值需要按下 “1”:

  1. top - 19:42:08 up 23:59,  2 users,  load average: 34.64, 35.80, 35.76
  2. Tasks: 679 total,   1 running, 678 sleeping,   0 stopped,   0 zombie
  3. Cpu0  : 29.5%us,  3.7%sy,  0.0%ni, 48.7%id, 17.9%wa,  0.0%hi,  0.1%si,  0.0%st
  4. Cpu1  : 29.3%us,  3.7%sy,  0.0%ni, 48.9%id, 17.9%wa,  0.0%hi,  0.1%si,  0.0%st
  5. Cpu2  : 26.1%us,  3.1%sy,  0.0%ni, 64.4%id,  6.0%wa,  0.0%hi,  0.3%si,  0.0%st
  6. Cpu3  : 25.9%us,  3.1%sy,  0.0%ni, 65.5%id,  5.4%wa,  0.0%hi,  0.1%si,  0.0%st
  7. Cpu4  : 24.9%us,  3.0%sy,  0.0%ni, 66.8%id,  5.0%wa,  0.0%hi,  0.3%si,  0.0%st
  8. Cpu5  : 24.9%us,  2.9%sy,  0.0%ni, 67.0%id,  4.8%wa,  0.0%hi,  0.3%si,  0.0%st
  9. Cpu6  : 24.2%us,  2.7%sy,  0.0%ni, 68.3%id,  4.5%wa,  0.0%hi,  0.3%si,  0.0%st
  10. Cpu7  : 24.3%us,  2.6%sy,  0.0%ni, 68.5%id,  4.2%wa,  0.0%hi,  0.3%si,  0.0%st
  11. Cpu8  : 23.8%us,  2.6%sy,  0.0%ni, 69.2%id,  4.1%wa,  0.0%hi,  0.3%si,  0.0%st
  12. Cpu9  : 23.9%us,  2.5%sy,  0.0%ni, 69.3%id,  4.0%wa,  0.0%hi,  0.3%si,  0.0%st
  13. Cpu10 : 23.3%us,  2.4%sy,  0.0%ni, 68.7%id,  5.6%wa,  0.0%hi,  0.0%si,  0.0%st
  14. Cpu11 : 23.3%us,  2.4%sy,  0.0%ni, 69.2%id,  5.1%wa,  0.0%hi,  0.0%si,  0.0%st
  15. Cpu12 : 21.8%us,  2.4%sy,  0.0%ni, 60.2%id, 15.5%wa,  0.0%hi,  0.0%si,  0.0%st
  16. Cpu13 : 21.9%us,  2.4%sy,  0.0%ni, 60.6%id, 15.2%wa,  0.0%hi,  0.0%si,  0.0%st
  17. Cpu14 : 21.4%us,  2.3%sy,  0.0%ni, 72.6%id,  3.7%wa,  0.0%hi,  0.0%si,  0.0%st
  18. Cpu15 : 21.5%us,  2.2%sy,  0.0%ni, 73.2%id,  3.1%wa,  0.0%hi,  0.0%si,  0.0%st
  19. Cpu16 : 21.2%us,  2.2%sy,  0.0%ni, 73.6%id,  3.0%wa,  0.0%hi,  0.0%si,  0.0%st
  20. Cpu17 : 21.2%us,  2.1%sy,  0.0%ni, 73.8%id,  2.8%wa,  0.0%hi,  0.0%si,  0.0%st
  21. Cpu18 : 20.9%us,  2.1%sy,  0.0%ni, 74.1%id,  2.9%wa,  0.0%hi,  0.0%si,  0.0%st
  22. Cpu19 : 21.0%us,  2.1%sy,  0.0%ni, 74.4%id,  2.5%wa,  0.0%hi,  0.0%si,  0.0%st
  23. Cpu20 : 20.7%us,  2.0%sy,  0.0%ni, 73.8%id,  3.4%wa,  0.0%hi,  0.0%si,  0.0%st
  24. Cpu21 : 20.8%us,  2.0%sy,  0.0%ni, 73.9%id,  3.2%wa,  0.0%hi,  0.0%si,  0.0%st
  25. Cpu22 : 20.8%us,  2.0%sy,  0.0%ni, 74.4%id,  2.8%wa,  0.0%hi,  0.0%si,  0.0%st
  26. Cpu23 : 20.8%us,  1.9%sy,  0.0%ni, 74.4%id,  2.8%wa,  0.0%hi,  0.0%si,  0.0%st
  27. Mem:  32865032k total, 30209248k used,  2655784k free,   370748k buffers
  28. Swap:  8388604k total,     5440k used,  8383164k free,  7986552k cached

wa 通常是 0%，如果经常在 1 之上，说明存储设备的速度已经太慢，无法跟上 cpu 的处理速度。

使用 atop 看下当前磁盘 IO 状态:

  1. ATOP - lemp              2017/01/23  19:42:32              ---------                10s elapsed
  2. PRC | sys    3.18s | user  33.24s | #proc    679 | #tslpu    28 | #zombie    0 | #exit      0 |
  3. CPU | sys      29% | user    330% | irq       1% | idle   1857% | wait    182% | curscal  69% |
  4. CPL | avg1   33.00 | avg5   35.29 | avg15  35.59 | csw    62610 | intr   76926 | numcpu    24 |
  5. MEM | tot    31.3G | free    2.1G | cache   7.6G | dirty  41.0M | buff  362.1M | slab    1.2G |
  6. SWP | tot     8.0G | free    8.0G |              |              | vmcom  23.9G | vmlim  23.7G |
  7. DSK |          sda | busy    100% | read       4 | write   1789 | MBw/s   2.84 | avio 5.58 ms |
  8. NET | transport    | tcpi   10357 | tcpo    9065 | udpi       0 | udpo       0 | tcpao    174 |
  9. NET | network      | ipi    10360 | ipo     9065 | ipfrw      0 | deliv  10359 | icmpo      0 |
  10. NET | eth0      4% | pcki    6649 | pcko    6136 | si 1478 Kbps | so 4115 Kbps | erro       0 |
  11. NET | lo      ---- | pcki    4082 | pcko    4082 | si 8967 Kbps | so 8967 Kbps | erro       0 |
  13.   PID   TID  THR  SYSCPU  USRCPU  VGROW  RGROW  RDDSK  WRDSK ST EXC S CPUNR  CPU CMD       1/12
  14.  9783     -  156   0.21s  19.44s     0K  -788K     4K  1344K --   - S     4 197% mysqld
  15.  5596     -    1   0.10s   0.62s 47204K 47004K     0K   220K --   - S    18   7% php-fpm
  16.  6429     -    1   0.06s   0.34s 19840K 19968K     0K     0K --   - S    21   4% php-fpm
  17.  6210     -    1   0.03s   0.30s -5216K -5204K     0K     0K --   - S    19   3% php-fpm
  18.  5757     -    1   0.05s   0.27s 26072K 26012K     0K     4K --   - S    13   3% php-fpm
  19.  6433     -    1   0.04s   0.28s -2816K -2816K     0K     0K --   - S    11   3% php-fpm
  20.  5846     -    1   0.06s   0.22s -2560K -2660K     0K     0K --   - S     7   3% php-fpm
  21.  5791     -    1   0.05s   0.21s  5764K  5692K     0K     0K --   - S    22   3% php-fpm
  22.  5860     -    1   0.04s   0.21s 48088K 47724K     0K     0K --   - S     1   3% php-fpm
  23.  6231     -    1   0.04s   0.20s  -256K    -4K     0K     0K --   - S     1   2% php-fpm
  24.  6154     -    1   0.03s   0.21s -3004K -3184K     0K     0K --   - S    21   2% php-fpm
  25.  6573     -    1   0.04s   0.20s  -512K  -168K     0K     0K --   - S     4   2% php-fpm
  26.  6435     -    1   0.04s   0.19s -3216K -2980K     0K     0K --   - S    15   2% php-fpm
  27.  5954     -    1   0.03s   0.20s     0K   164K     0K     4K --   - S     0   2% php-fpm
  28.  6133     -    1   0.03s   0.19s 41056K 40432K     0K     0K --   - S    18   2% php-fpm
  29.  6132     -    1   0.02s   0.20s 37836K 37440K     0K     0K --   - S    11   2% php-fpm
  30.  6242     -    1   0.03s   0.19s -12.2M -12.3M     0K     4K --   - S    12   2% php-fpm
  31.  6285     -    1   0.02s   0.19s 39516K 39420K     0K     0K --   - S     3   2% php-fpm
  32.  6455     -    1   0.05s   0.16s 29008K 28560K     0K     0K --   - S    14   2% php-fpm

在本例中磁盘 sda 已经 100% busy，已经严重达到性能瓶颈。按 ‘d’ 看下是哪些进程在使用磁盘IO:

  1. ATOP - lemp               2017/01/23  19:42:46               ---------               2s elapsed
  2. PRC | sys    0.24s | user   1.99s | #proc    679 | #tslpu    54 | #zombie    0 | #exit      0 |
  3. CPU | sys      11% | user    101% | irq       1% | idle   2089% | wait    208% | curscal  63% |
  4. CPL | avg1   38.49 | avg5   36.48 | avg15  35.98 | csw     4654 | intr    6876 | numcpu    24 |
  5. MEM | tot    31.3G | free    2.2G | cache   7.6G | dirty  48.7M | buff  362.1M | slab    1.2G |
  6. SWP | tot     8.0G | free    8.0G |              |              | vmcom  23.9G | vmlim  23.7G |
  7. DSK |          sda | busy    100% | read       2 | write    362 | MBw/s   2.28 | avio 5.49 ms |
  8. NET | transport    | tcpi    1031 | tcpo     968 | udpi       0 | udpo       0 | tcpao     45 |
  9. NET | network      | ipi     1031 | ipo      968 | ipfrw      0 | deliv   1031 | icmpo      0 |
  10. NET | eth0      1% | pcki     558 | pcko     508 | si  762 Kbps | so 1077 Kbps | erro       0 |
  11. NET | lo      ---- | pcki     406 | pcko     406 | si 2273 Kbps | so 2273 Kbps | erro       0 |
  13.   PID          TID         RDDSK         WRDSK        WCANCL         DSK        CMD         1/5
  14.  9783            -            0K          468K           16K         40%        mysqld
  15.  1930            -            0K          212K            0K         18%        flush-8:0
  16.  5896            -            0K          152K            0K         13%        nginx
  17.   880            -            0K          148K            0K         13%        jbd2/sda5-8
  18.  5909            -            0K           60K            0K          5%        nginx
  19.  5906            -            0K           36K            0K          3%        nginx
  20.  5907            -           16K            8K            0K          2%        nginx
  21.  5903            -           20K            0K            0K          2%        nginx
  22.  5901            -            0K           12K            0K          1%        nginx
  23.  5908            -            0K            8K            0K          1%        nginx
  24.  5894            -            0K            8K            0K          1%        nginx
  25.  5911            -            0K            8K            0K          1%        nginx
  26.  5900            -            0K            4K            4K          0%        nginx
  27.  5551            -            0K            4K            0K          0%        php-fpm
  28.  5913            -            0K            4K            0K          0%        nginx
  29.  5895            -            0K            4K            0K          0%        nginx
  30.  6133            -            0K            0K            0K          0%        php-fpm
  31.  5780            -            0K            0K            0K          0%        php-fpm
  32.  6675            -            0K            0K            0K          0%        atop

也可以使用 iotop -oPa 查看哪些进程占用磁盘 IO:

  1. Total DISK READ: 15.02 K/s | Total DISK WRITE: 3.82 M/s
  2.   PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
  3.  1930 be/4 root          0.00 B   1956.00 K  0.00 % 83.34 % [flush-8:0]
  4.  5914 be/4 nginx         0.00 B      0.00 B  0.00 % 36.56 % nginx: cache manager process
  5.   880 be/3 root          0.00 B     21.27 M  0.00 % 35.03 % [jbd2/sda5-8]
  6.  5913 be/2 nginx        36.00 K   1000.00 K  0.00 %  8.94 % nginx: worker process
  7.  5910 be/2 nginx         0.00 B   1048.00 K  0.00 %  8.43 % nginx: worker process
  8.  5896 be/2 nginx        56.00 K    452.00 K  0.00 %  6.91 % nginx: worker process
  9.  5909 be/2 nginx        20.00 K   1144.00 K  0.00 %  6.24 % nginx: worker process
  10.  5890 be/2 nginx        48.00 K    692.00 K  0.00 %  6.07 % nginx: worker process
  11.  5892 be/2 nginx        84.00 K    736.00 K  0.00 %  5.71 % nginx: worker process
  12.  5901 be/2 nginx        20.00 K    504.00 K  0.00 %  5.46 % nginx: worker process
  13.  5899 be/2 nginx         0.00 B    596.00 K  0.00 %  5.14 % nginx: worker process
  14.  5897 be/2 nginx        28.00 K   1388.00 K  0.00 %  4.90 % nginx: worker process
  15.  5908 be/2 nginx        48.00 K    700.00 K  0.00 %  4.43 % nginx: worker process
  16.  5905 be/2 nginx        32.00 K   1140.00 K  0.00 %  4.36 % nginx: worker process
  17.  5900 be/2 nginx         0.00 B   1208.00 K  0.00 %  4.31 % nginx: worker process
  18.  5904 be/2 nginx        36.00 K   1244.00 K  0.00 %  2.80 % nginx: worker process
  19.  5895 be/2 nginx        16.00 K    780.00 K  0.00 %  2.50 % nginx: worker process
  20.  5907 be/2 nginx         0.00 B   1548.00 K  0.00 %  2.43 % nginx: worker process
  21.  5903 be/2 nginx        36.00 K   1032.00 K  0.00 %  2.34 % nginx: worker process
  22.  6130 be/4 nginx         0.00 B     72.00 K  0.00 %  2.18 % php-fpm: pool www
  23.  5906 be/2 nginx        12.00 K    844.00 K  0.00 %  2.10 % nginx: worker process
  24.  5889 be/2 nginx        40.00 K   1164.00 K  0.00 %  2.00 % nginx: worker process
  25.  5894 be/2 nginx        44.00 K    760.00 K  0.00 %  1.61 % nginx: worker process
  26.  5902 be/2 nginx        52.00 K    992.00 K  0.00 %  1.55 % nginx: worker process
  27.  5893 be/2 nginx        64.00 K    972.00 K  0.00 %  1.22 % nginx: worker process
  28.  5814 be/4 nginx        36.00 K     44.00 K  0.00 %  1.06 % php-fpm: pool www
  29.  6159 be/4 nginx         4.00 K      4.00 K  0.00 %  1.00 % php-fpm: pool www
  30.  5693 be/4 nginx         0.00 B      4.00 K  0.00 %  0.86 % php-fpm: pool www
  31.  5912 be/2 nginx        68.00 K    300.00 K  0.00 %  0.72 % nginx: worker process
  32.  5911 be/2 nginx        20.00 K    788.00 K  0.00 %  0.72 % nginx: worker process

通过 man iotop 可以看下这几个参数的含义：

  1. -o, --only
  2.        Only show processes or threads actually doing I/O, instead of showing all processes or threads. This can be dynamically toggled by pressing o.
  3. -P, --processes
  4.        Only show processes. Normally iotop shows all threads.
  6. -a, --accumulated
  7.        Show accumulated I/O instead of bandwidth. In this mode, iotop shows the amount of I/O processes have done since iotop started.

节点上部署了其它非 K8S 管理的服务

TODO 优化

比如在节点上装了数据库，但不被 K8S 所管理，这是用法不正确，不建议在 K8S 节点上部署其它进程。

参考资料