[Linux] Network stalls
Peter Capazzi
linux@flux.org
Thu, 19 Jun 2008 11:15:21 -0400
Thank you Kwan, Mike and Adam.
That scp to /dev/null was a great idea. The transfers seemed to be
consistent or at least recovered well if they dipped.=20
When I copied the 15 GB file again (to a file, not /dev/null) I used
vmstat to analyze. The transfer rate dropped to about 15 MB/sec where
the heading re-prints. It looks like the io's blockout is consistent
during this period... But the sys looks like it's having trouble and
then the stats drop when the rate changes.=20
The next time I tried analyzing with iostat... As soon as I started the
copy it dropped from 32->15 Mb/sec.=20
I'm new to this analysis and attempts at tweaking so bear with me... =
=3D)
Below are two items, output from vmstat, then iostat:
1 2 64 16616 45176 7671988 0 0 0 12096 9891 27372 11
6 58 25 0
1 3 64 15116 45240 7671924 0 0 0 14340 10518 26767 12
7 43 38 0
1 5 64 17004 45240 7670896 0 0 0 9576 12294 22357 13
7 31 49 0
1 4 64 15380 45240 7671924 0 0 0 12940 10514 26669 11
5 34 50 0
0 7 64 17596 45240 7669868 0 0 0 9464 12257 22320 12
7 44 36 0
2 2 64 16148 45240 7670896 0 0 0 17976 9734 27508 11
7 39 43 0
0 4 64 18004 45240 7668840 0 0 0 12656 12229 17764 11
7 42 39 0
1 3 64 16876 45240 7668840 0 0 0 14448 9386 29101 13
7 33 47 0
0 4 64 16172 45240 7668840 0 0 0 12936 9988 27351 11
6 50 33 0
1 3 64 14912 45252 7670884 0 0 0 11600 10803 26146 12
7 33 48 0
1 5 64 16808 45244 7667808 0 0 0 13384 12212 22697 11
7 32 49 0
1 6 64 15788 45236 7669872 0 0 0 12052 10825 25866 11
8 32 49 0
0 6 64 14968 45236 7669872 0 0 0 11368 10645 26205 12
8 31 50 0
1 5 64 17160 45236 7667816 0 0 0 14568 11613 24319 11
7 43 39 0
1 4 64 16332 45236 7668844 0 0 0 8012 10669 26260 12
7 40 41 0
1 9 64 16364 45236 7668844 0 0 0 19216 11526 23776 10
7 56 26 0
0 9 64 15668 45236 7668844 0 0 0 9520 10410 26565 12
8 55 25 0
0 10 64 16592 45236 7667816 0 0 0 16472 10867 23439 11
7 56 25 0
0 9 64 16180 45236 7667816 0 0 0 12144 1456 3469 1 1
74 24 0
procs -----------memory---------- ---swap-- -----io---- -system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy
id wa st
1 9 64 17324 45236 7667816 0 0 0 12440 6273 14221 8
7 63 22 0
0 8 64 17508 45236 7667816 0 0 0 12320 2185 4760 3 2
71 24 0
0 8 64 18184 45236 7668844 0 0 0 14164 4426 10456 5
4 57 34 0
1 4 64 15868 45288 7669820 0 0 0 13940 4094 6822 6 3
48 42 0
0 8 64 18028 45288 7669820 0 0 0 12324 4712 12785 6
4 42 48 0
1 8 64 15112 45288 7670848 0 0 0 12320 3157 6009 4 3
44 49 0
0 9 64 15848 45288 7671876 0 0 0 9088 2021 5499 3 1
71 25 0
0 2 64 18308 45288 7668792 0 0 0 13144 4688 9961 7 5
53 35 0
0 5 64 15916 45288 7670848 0 0 0 12392 5912 12519 7
4 42 48 0
1 7 64 18344 45288 7668792 0 0 0 12320 3623 9137 4 3
44 49 0
1 7 64 15992 45284 7670852 0 0 0 12320 2097 4311 2 2
47 49 0
0 8 64 15624 45284 7670852 0 0 0 13028 4765 10238 6
4 42 48 0
0 4 64 17148 45304 7668776 0 0 0 15008 4706 12598 6
5 44 45 0
5 4 64 16768 45304 7668776 0 0 0 9392 2438 5054 4 2
48 46 0
1 5 64 18016 45304 7668776 0 0 0 14004 5498 11330 7
5 40 47 0
0 2 64 16728 45304 7669804 0 0 0 16640 3693 8632 5 3
58 33 0
1 4 64 16632 45304 7669804 0 0 0 11532 4392 7967 7 4
66 23 0
1 4 64 17136 45304 7669804 0 0 0 12320 4294 9348 7 5
66 22 0
avg-cpu: %user %nice %system %iowait %steal %idle
0.40 0.00 0.27 2.48 0.00 96.85
11.00 0.00 5.50 0.00 0.00 83.75
11.00 0.00 6.00 0.00 0.00 83.25
12.00 0.00 5.50 0.75 0.00 81.75
12.13 0.00 4.95 0.00 0.00 82.43
7.25 0.00 5.00 8.50 0.00 79.50
5.00 0.00 2.50 15.75 0.00 76.50
1.75 0.00 2.00 22.25 0.00 74.50
5.20 0.00 2.72 19.31 0.00 72.03
5.00 0.00 4.75 14.25 0.00 76.50
4.00 0.00 2.25 19.00 0.00 75.00
3.47 0.00 1.98 18.32 0.00 75.74
4.00 0.00 3.25 16.25 0.00 76.75
5.20 0.00 2.23 17.08 0.00 75.00
4.25 0.00 3.75 17.50 0.00 75.00
3.50 0.00 1.75 18.50 0.00 76.75
12.62 0.00 5.94 16.09 0.00 64.36
1.00 0.00 1.25 24.50 0.00 74.25
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0.97 165.66 0.32 8.26 39.09 2782.74 19.55 1391.37
328.68 7.18 836.43 6.80 5.84
cciss/c0d0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
cciss/c0d0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
cciss/c0d0 0.00 48.00 0.00 5.00 0.00 848.00 0.00 424.00
169.60 0.06 12.80 8.80 4.40
cciss/c0d0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
cciss/c0d0 0.00 1982.00 2.00 45.00 32.00 16576.00 16.00 8288.00
353.36 22.81 316.17 10.89 51.20
cciss/c0d0 0.00 2974.00 3.00 90.00 48.00 33152.00 24.00 16576.00
356.99 55.36 316.69 9.46 88.00
cciss/c0d0 0.00 1983.00 1.00 89.00 16.00 33152.00 8.00 16576.00
368.53 80.82 645.51 10.09 90.80
cciss/c0d0 0.00 1972.28 3.96 179.21 63.37 65837.62 31.68
32918.81 359.78 31.29 476.71 3.78 69.31
cciss/c0d0 0.00 3966.00 3.00 89.00 48.00 33152.00 24.00 16576.00
360.87 70.79 631.74 8.78 80.80
cciss/c0d0 0.00 1983.00 2.00 89.00 32.00 33152.00 16.00 16576.00
364.66 78.10 637.71 9.71 88.40
cciss/c0d0 0.00 1962.38 2.97 177.23 47.52 65647.52 23.76
32823.76 364.57 54.07 479.93 5.16 93.07
cciss/c0d0 0.00 2973.00 2.00 92.00 32.00 33664.00 16.00 16832.00
358.47 40.64 310.68 9.96 93.60
cciss/c0d0 0.00 2952.48 2.97 132.67 47.52 48744.55 23.76
24372.28 359.71 40.77 319.80 6.69 90.69
cciss/c0d0 0.00 2973.00 3.00 137.00 48.00 49904.00 24.00
24952.00 356.80 37.92 312.31 6.49 90.80
cciss/c0d0 0.00 1984.00 3.00 133.00 48.00 49728.00 24.00
24864.00 366.00 39.51 312.56 6.71 91.20
cciss/c0d0 0.00 3939.60 0.00 65.35 0.00 24348.51 0.00 12174.26
372.61 97.33 779.15 14.67 95.84
cciss/c0d0 0.00 3402.00 0.00 154.00 0.00 57168.00 0.00
28584.00 371.22 113.02 756.83 6.52 100.40