Difference: SuperComputing2012 (11 vs. 12)

Revision 122012-11-06 - igable

Line: 1 to 1

SuperComputing 2012

Line: 416 to 416

After tcp_syncookies fix, back to kernel 2.6.32 (November 5)

Summary: We are now achieving very stable reads at 38 Gbit/s. Ramiro found that "net.ipv4.tcp_syncookies=0" must be set it order to avoid stuck sockets. After trying a large number of configuration variation we haven't achieved a write speed above 24 Gbit/s. Hyper Threading was disabled on the both send and recieve hosts.

The sockets getting stuck problem was solved completely with:

%STARTCONSOLE% net.ipv4.tcp_sack=1 net.ipv4.tcp_syncookies=0 net.ipv4.tcp_no_metrics_save=1 %ENDCONSOLE%

'net.ipv4.tcp_syncookies=0' MUST be set or you will get stuck sockets.

  configured with 7 x (2 disk virtual disk reader side)
Line: 445 to 457
  Exit Status: OK %ENDCONSOLE%
Now back to 15 disk to try out the high number of streams

%STARTCONSOLE% 05/11 17:28:55 Net In: 4.288 Gb/s Avg: 20.925 Gb/s 100.00% ( 00s )

FDTWriterSession ( c0ba0ef1-9c62-4133-8a94-e264edec70e3 ) final stats: Started: Mon Nov 05 17:22:04 PST 2012 Ended: Mon Nov 05 17:29:00 PST 2012 Transfer period: 06m 56s TotalBytes: 1072723868000 TotalNetworkBytes: 1072723868000 Exit Status: OK %ENDCONSOLE%

Now trying software raid. Looks to be limited to about 10 Gbit/s with one java thread pegged at 100%: %STARTCONSOLE% mdadm --create --verbose /dev/md0 --level=0 --raid-devices=15 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp mkfs.xfs -f -d sunit=1024,swidth=1024 /dev/md0 05/11 18:04:36 Net In: 8.790 Gb/s Avg: 9.880 Gb/s 14.80% %ENDCONSOLE%

Double check we are still doing good reads. Very stable at 38 Gbit/s:

%STARTCONSOLE% 05/11 18:24:54 Net Out: 38.457 Gb/s Avg: 38.400 Gb/s 38.38% ( 03m 26s ) %ENDCONSOLE%

META FILEATTACHMENT attachment="transfer-notes-2012-11-01.png" attr="" comment="" date="1351841497" name="transfer-notes-2012-11-01.png" path="transfer-notes-2012-11-01.png" size="128899" user="igable" version="1"
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback