Difference: SuperComputing2012 (1 vs. 12)

Revision 122012-11-06 - igable

Line: 1 to 1
 

SuperComputing 2012

Line: 416 to 416
 

After tcp_syncookies fix, back to kernel 2.6.32 (November 5)

Added:
>
>
Summary: We are now achieving very stable reads at 38 Gbit/s. Ramiro found that "net.ipv4.tcp_syncookies=0" must be set it order to avoid stuck sockets. After trying a large number of configuration variation we haven't achieved a write speed above 24 Gbit/s. Hyper Threading was disabled on the both send and recieve hosts.

The sockets getting stuck problem was solved completely with:

%STARTCONSOLE% net.ipv4.tcp_sack=1 net.ipv4.tcp_syncookies=0 net.ipv4.tcp_no_metrics_save=1 %ENDCONSOLE%

'net.ipv4.tcp_syncookies=0' MUST be set or you will get stuck sockets.

  configured with 7 x (2 disk virtual disk reader side)
Line: 445 to 457
  Exit Status: OK %ENDCONSOLE%
Added:
>
>
Now back to 15 disk to try out the high number of streams

%STARTCONSOLE% 05/11 17:28:55 Net In: 4.288 Gb/s Avg: 20.925 Gb/s 100.00% ( 00s )

FDTWriterSession ( c0ba0ef1-9c62-4133-8a94-e264edec70e3 ) final stats: Started: Mon Nov 05 17:22:04 PST 2012 Ended: Mon Nov 05 17:29:00 PST 2012 Transfer period: 06m 56s TotalBytes: 1072723868000 TotalNetworkBytes: 1072723868000 Exit Status: OK %ENDCONSOLE%

Now trying software raid. Looks to be limited to about 10 Gbit/s with one java thread pegged at 100%: %STARTCONSOLE% mdadm --create --verbose /dev/md0 --level=0 --raid-devices=15 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp mkfs.xfs -f -d sunit=1024,swidth=1024 /dev/md0 05/11 18:04:36 Net In: 8.790 Gb/s Avg: 9.880 Gb/s 14.80% %ENDCONSOLE%

Double check we are still doing good reads. Very stable at 38 Gbit/s:

%STARTCONSOLE% 05/11 18:24:54 Net Out: 38.457 Gb/s Avg: 38.400 Gb/s 38.38% ( 03m 26s ) %ENDCONSOLE%

 
META FILEATTACHMENT attachment="transfer-notes-2012-11-01.png" attr="" comment="" date="1351841497" name="transfer-notes-2012-11-01.png" path="transfer-notes-2012-11-01.png" size="128899" user="igable" version="1"

Revision 112012-11-06 - igable

Line: 1 to 1
 

SuperComputing 2012

Line: 414 to 414
  Exit Status: OK %ENDCONSOLE%
Added:
>
>

After tcp_syncookies fix, back to kernel 2.6.32 (November 5)

configured with 7 x (2 disk virtual disk reader side)

%STARTCONSOLE% [root@sc04 ~]# fdtClient -P 20 -c 10.20.3.101 -fl slotmount -d / 05/11 15:33:59 Net In: 0.000 b/s Avg: 24.743 Gb/s 100.00% ( 00s )

FDTWriterSession ( 2773bed5-4842-479c-8cd5-845d562b9429 ) final stats: Started: Mon Nov 05 15:25:54 PST 2012 Ended: Mon Nov 05 15:34:02 PST 2012 Transfer period: 08m 07s TotalBytes: 1501813415200 TotalNetworkBytes: 1501813415200 Exit Status: OK %ENDCONSOLE%

%STARTCONSOLE% [root@sc04 ~]# fdtClient -P 40 -c 10.20.3.101 -fl slotmount -d / 5/11 15:47:10 Net Out: 0.000 b/s Avg: 24.617 Gb/s 100.00% ( 00s )

FDTReaderSession ( 499258af-7340-47d4-b318-8e1f128a4747 ) final stats: Started: Mon Nov 05 15:39:00 PST 2012 Ended: Mon Nov 05 15:47:11 PST 2012 Transfer period: 08m 10s TotalBytes: 1501813415200 TotalNetworkBytes: 1501813415200 Exit Status: OK %ENDCONSOLE%

 
META FILEATTACHMENT attachment="transfer-notes-2012-11-01.png" attr="" comment="" date="1351841497" name="transfer-notes-2012-11-01.png" path="transfer-notes-2012-11-01.png" size="128899" user="igable" version="1"

Revision 102012-11-05 - igable

Line: 1 to 1
 

SuperComputing 2012

Line: 341 to 341
 tcp 14390168 0 ::ffff:10.20.3.101:54321 ::ffff:10.20.3.104:58235 ESTABLISHED 5898/java %ENDCONSOLE%
Changed:
<
<

Testing with drive subsets ( Nov 4)

>
>

Kernel 3.2 testing ,and independent raid testing (Nov 3-4).

 
Changed:
<
<
Testing the first 7 drives we still finish with things in the sent-q (4 streams).
>
>
Testing the first 7 drives we still finish with things in the send-q (4 streams).
 %STARTCONSOLE% tcp 0 0 ::ffff:10.20.3.104:35078 ::ffff:10.20.3.101:54321 ESTABLISHED 3663/java tcp 0 11200000 ::ffff:10.20.3.104:35081 ::ffff:10.20.3.101:54321 ESTABLISHED 3663/java tcp 0 13251840 ::ffff:10.20.3.104:35080 ::ffff:10.20.3.101:54321 ESTABLISHED 3663/java %ENDCONSOLE%
Added:
>
>
2 streams looks to work reliably.
 
Added:
>
>
Testing with 3.2.32 kernel from shawn we see some interesting behavior with mem to mem right after the start then it stabalizes at ~38:

%STARTCONSOLE% 04/11 20:04:53 Net In: 39.153 Gb/s Avg: 39.153 Gb/s 04/11 20:04:58 Net In: 38.660 Gb/s Avg: 38.906 Gb/s 04/11 20:05:03 Net In: 38.836 Gb/s Avg: 38.883 Gb/s 04/11 20:05:08 Net In: 34.049 Gb/s Avg: 37.673 Gb/s 04/11 20:05:13 Net In: 39.013 Gb/s Avg: 37.939 Gb/s 04/11 20:05:18 Net In: 25.254 Gb/s Avg: 35.824 Gb/s 04/11 20:05:23 Net In: 13.737 Gb/s Avg: 32.669 Gb/s 04/11 20:05:28 Net In: 30.079 Gb/s Avg: 32.345 Gb/s 04/11 20:05:33 Net In: 34.175 Gb/s Avg: 32.547 Gb/s 04/11 20:05:38 Net In: 38.890 Gb/s Avg: 33.181 Gb/s 04/11 20:05:43 Net In: 23.152 Gb/s Avg: 32.269 Gb/s 04/11 20:05:48 Net In: 35.668 Gb/s Avg: 32.552 Gb/s %ENDCONSOLE%

Disk to disk with 7 disks we see pretty wild fluctuations in the throughput, but no sockets get stuck (with 4 streams):

%STARTCONSOLE% 04/11 20:21:44 Net In: 0.000 b/s Avg: 12.011 Gb/s 100.00% ( 00s )

FDTWriterSession ( bdb2c737-6d7d-41cf-bd3d-3b1ceed254ea ) final stats: Started: Sun Nov 04 20:13:23 PST 2012 Ended: Sun Nov 04 20:21:46 PST 2012 Transfer period: 08m 22s TotalBytes: 750906707600 TotalNetworkBytes: 750906707600 Exit Status: OK %ENDCONSOLE%

Repeat the above with 10 streams much better (using 7 disks same raid controller, still some fluctuations):

Complete log at: https://gist.github.com/0bc3087b37ea26cff999 %STARTCONSOLE% 04/11 21:11:12 Net Out: 3.376 Gb/s Avg: 19.598 Gb/s 100.00% ( 00s )

FDTReaderSession ( 26051cb0-ebe3-4000-86ef-2bb23c9e62f2 ) final stats: Started: Sun Nov 04 21:05:22 PST 2012 Ended: Sun Nov 04 21:11:17 PST 2012 Transfer period: 05m 54s TotalBytes: 858179094400 TotalNetworkBytes: 858179094400 Exit Status: OK %ENDCONSOLE%

Very nice Stable reads with the 3.2.32 kernel: %STARTCONSOLE% 04/11 22:05:03 Net Out: 1.515 Gb/s Avg: 35.881 Gb/s 100.00% ( 00s )

FDTReaderSession ( f5ca4066-2fa0-4a8a-9ab7-374c1aaaeed3 ) final stats: Started: Sun Nov 04 22:00:37 PST 2012 Ended: Sun Nov 04 22:05:03 PST 2012 Transfer period: 04m 25s TotalBytes: 1179996254800 TotalNetworkBytes: 1179996254800 Exit Status: OK %ENDCONSOLE%

 
META FILEATTACHMENT attachment="transfer-notes-2012-11-01.png" attr="" comment="" date="1351841497" name="transfer-notes-2012-11-01.png" path="transfer-notes-2012-11-01.png" size="128899" user="igable" version="1"

Revision 92012-11-05 - igable

Line: 1 to 1
 

SuperComputing 2012

Line: 341 to 341
 tcp 14390168 0 ::ffff:10.20.3.101:54321 ::ffff:10.20.3.104:58235 ESTABLISHED 5898/java %ENDCONSOLE%
Added:
>
>

Testing with drive subsets ( Nov 4)

Testing the first 7 drives we still finish with things in the sent-q (4 streams). %STARTCONSOLE% tcp 0 0 ::ffff:10.20.3.104:35078 ::ffff:10.20.3.101:54321 ESTABLISHED 3663/java tcp 0 11200000 ::ffff:10.20.3.104:35081 ::ffff:10.20.3.101:54321 ESTABLISHED 3663/java tcp 0 13251840 ::ffff:10.20.3.104:35080 ::ffff:10.20.3.101:54321 ESTABLISHED 3663/java %ENDCONSOLE%

 

META FILEATTACHMENT attachment="transfer-notes-2012-11-01.png" attr="" comment="" date="1351841497" name="transfer-notes-2012-11-01.png" path="transfer-notes-2012-11-01.png" size="128899" user="igable" version="1"

Revision 82012-11-02 - igable

Line: 1 to 1
 

SuperComputing 2012

Line: 7 to 7
  No problems with memoty to memory: 37.806 Gb/s
Changed:
<
<
>
>
%STARTCONSOLE%
 [root@sc02 ~]# fdtClient -P 7 -c 192.168.100.4 /dev/zero -d /dev/null Oct 28, 2012 11:49:59 PM lia.util.net.common.Config INFO: Using lia.util.net.copy.PosixFSFileChannelProviderFactory as FileChannelProviderFactory
Line: 19 to 19
 28/10 23:50:14 Net Out: 38.128 Gb/s Avg: 38.129 Gb/s 28/10 23:50:19 Net Out: 36.775 Gb/s Avg: 37.678 Gb/s 28/10 23:50:24 Net Out: 38.197 Gb/s Avg: 37.806 Gb/s
Changed:
<
<
>
>
%ENDCONSOLE%
 

Disk to Disk within a machine

Changed:
<
<
>
>
%STARTCONSOLE%
 MegaCli -CfgLdAdd -r0 [252:1] WT NORA DIRECT -strpsz 1024 -a0 [root@sc02 ~]# fdtCopy if=/ssd1/010Gfile_n0010.dat of=/ssd7/010Gfile_n0010.dat [Sun Oct 28 20:56:57 PDT 2012] Current Speed = 441.648 MB/s Avg Speed: 436.479 MB/s Total Transfer: 4.277 GB
Changed:
<
<
>
>
%ENDCONSOLE%
 

Network transfer with 7 virtual disks and 1 fdtServers, 7 parallel streams

Grand total with 7 disks, 1 FDT Server: 10.533 Gb/s

Changed:
<
<
>
>
%STARTCONSOLE%
 [root@sc02 ssd6]# fdtClient -P 7 -c 192.168.100.4 -fl /ssd6/filelist.txt -d / Avg: 10.533 Gb/s 100.00% ( 00s ) FDTReaderSession ( 13f62162-4fb5-41c5-b4fb-db9af9c453f3 ) final stats:
Line: 43 to 43
  TotalBytes: 751619276800 TotalNetworkBytes: 751619276800 Exit Status: OK
Changed:
<
<
>
>
%ENDCONSOLE%
 

Network transfer with 7 virtual disks and 7 fdtServers

Line: 52 to 52
 Grand total with 7 disks, 7 FDT Servers: 22.196 Gb/s
Changed:
<
<
>
>
%STARTCONSOLE%
 fdtClient -c 192.168.100.4 -p 54321 -d /ssd1/ /ssd1/100Gfile_x.dat > /root/fdt1.log & fdtClient -c 192.168.100.4 -p 54322 -d /ssd2/ /ssd2/100Gfile_x.dat > /root/fdt2.log & fdtClient -c 192.168.100.4 -p 54323 -d /ssd3/ /ssd3/100Gfile_x.dat > /root/fdt3.log &
Line: 60 to 60
 fdtClient -c 192.168.100.4 -p 54325 -d /ssd5/ /ssd5/100Gfile_x.dat > /root/fdt5.log & fdtClient -c 192.168.100.4 -p 54326 -d /ssd6/ /ssd6/100Gfile_x.dat > /root/fdt6.log & fdtClient -c 192.168.100.4 -p 54327 -d /ssd7/ /ssd7/100Gfile_x.dat > /root/fdt7.log &
Changed:
<
<
>
>
%ENDCONSOLE%
 
Changed:
<
<
>
>
%STARTCONSOLE%
 Avg: 3.067 Gb/s 100.00% ( 00s ) FDTWriterSession ( 9920c20c-6378-4ef1-b14b-8393b5c1eafb ) final stats: Started: Sun Oct 28 22:57:04 PDT 2012
Line: 125 to 125
  TotalBytes: 107374182400 TotalNetworkBytes: 107374182400 Exit Status: OK
Changed:
<
<
>
>
%ENDCONSOLE%
 

7 clients one server

Changed:
<
<
>
>
%STARTCONSOLE%
 9cee52db-3569-4b29-b894-06a536c14409Net In: 1.246 Gb/s Avg: 1.216 Gb/s 99.14% ( 06s ) 9ea1e5bc-9c3c-41c0-8772-0e9a8845cbd5Net In: 1.238 Gb/s Avg: 1.219 Gb/s 99.38% ( 04s ) 9f1983ee-585f-47be-9b76-5d6d6d51f688Net In: 1.226 Gb/s Avg: 1.217 Gb/s 99.21% ( 05s )
Line: 264 to 264
 INFO: [ FDTWriterSession ] Post Processing started Oct 29, 2012 10:48:19 AM lia.util.net.copy.FDTWriterSession doPostProcessing INFO: [ FDTWriterSession ] No post processing filters defined/processed.
Changed:
<
<
>
>
%ENDCONSOLE%
 

Post firmware single clients (November 1)

Line: 276 to 275
 Details:

With a 15 writer server halfway though the transfer the speed was cut in half.

Changed:
<
<
>
>
%STARTCONSOLE%
 01/11 20:16:34 Net In: 11.824 Gb/s Avg: 16.476 Gb/s 99.90% ( 00s )
Changed:
<
<
>
>
%ENDCONSOLE%
  After the transfer of the files was complete FDT continued to report line like such:
Changed:
<
<
>
>
%STARTCONSOLE%
 01/11 20:16:30 Net Out: 11.686 Gb/s Avg: 16.490 Gb/s 99.54% ( 03s ) 01/11 20:16:35 Net Out: 11.770 Gb/s Avg: 16.459 Gb/s 100.00% ( 00s ) 01/11 20:16:40 Net Out: 0.000 b/s Avg: 16.354 Gb/s 100.00% ( 00s )
Line: 290 to 289
 01/11 20:16:50 Net Out: 0.000 b/s Avg: 16.147 Gb/s 100.00% ( 00s ) 01/11 20:16:55 Net Out: 0.000 b/s Avg: 16.045 Gb/s 100.00% ( 00s ) 01/11 20:17:00 Net Out: 0.000 b/s Avg: 15.945 Gb/s 100.00% ( 00s )
Changed:
<
<
>
>
%ENDCONSOLE%
  until eventually killed.

The problem with half open connection was eliminated with the firmware upgrade:

Changed:
<
<
>
>
%STARTCONSOLE%
 tcp 0 1 ::ffff:10.20.3.104:54008 ::ffff:10.20.3.101:54321 SYN_SENT 5684/java
Changed:
<
<
>
>
%ENDCONSOLE%
 was fixed by moving to the 2.11.500 firmware version for the Mellanox CX3.

Using a single writer thread the transfer was completely stable.

Changed:
<
<
>
>
%STARTCONSOLE%
 01/11 22:33:59 Net In: 2.475 Gb/s Avg: 10.954 Gb/s 100.00% ( 00s )

FDTWriterSession ( ff0ff011-80f1-4497-8397-72e1a18ab78b ) final stats:

Line: 313 to 312
  TotalBytes: 1609085802000 TotalNetworkBytes: 1609085802000 Exit Status: OK
Changed:
<
<
>
>
%ENDCONSOLE%
  Using 2 writer threads also completely stable:
Changed:
<
<
>
>
%STARTCONSOLE%
 01/11 23:04:14 Net In: 3.423 Gb/s Avg: 21.328 Gb/s 100.00% ( 00s )

FDTWriterSession ( d9cad4f0-8987-4079-8ec0-7e8f73890afb ) final stats:

Line: 327 to 326
  TotalBytes: 1609085802000 TotalNetworkBytes: 1609085802000 Exit Status: OK
Changed:
<
<
>
>
%ENDCONSOLE%
  Using 3 writers there was a problem. Part way through the transfer the first 4 connection stop doing anything.
Changed:
<
<
>
>
%STARTCONSOLE%
 01/11 23:31:07 Net In: 6.523 Gb/s Avg: 19.177 Gb/s 100.00% ( 00s )

netstat

Line: 341 to 339
 tcp 0 0 ::ffff:10.20.3.101:54321 ::ffff:10.20.3.104:58234 ESTABLISHED 5898/java tcp 0 0 ::ffff:10.20.3.101:54321 ::ffff:10.20.3.104:58237 ESTABLISHED 5898/java tcp 14390168 0 ::ffff:10.20.3.101:54321 ::ffff:10.20.3.104:58235 ESTABLISHED 5898/java
Changed:
<
<
>
>
%ENDCONSOLE%
 

Revision 62012-11-02 - igable

Line: 1 to 1
 

SuperComputing 2012

Line: 267 to 267
 
Changed:
<
<

Post firmware single clients

>
>

Post firmware single clients (November 1)

 
Changed:
<
<
Halfway though the transfer the speed was cut in half.
>
>
Summary: After a firmware upgrade the card no longer has connections getting stuck half open ( SYN_SENT ). Writing to disk using the -wCount feature works however we run into problems with more then 2 writer threads. Prior to the firmware update, when we hit these problems the machine would require a complete reboot in order to get back connectivity on the interface. However now we see what looks like fewer threads finish the job. Is this a FDT application problem or are we still seeing issues with the card?

transfer-notes-2012-11-01.png

Details:

With a 15 writer server halfway though the transfer the speed was cut in half.

 
01/11 20:16:34   Net In: 11.824 Gb/s   Avg: 16.476 Gb/s 99.90% ( 00s )
Line: 321 to 328
  TotalNetworkBytes: 1609085802000 Exit Status: OK
Added:
>
>
Using 3 writers there was a problem. Part way through the transfer the first 4 connection stop doing anything.

01/11 23:31:07   Net In: 6.523 Gb/s   Avg: 19.177 Gb/s 100.00% ( 00s )

netstat
tcp        0      0 ::ffff:10.20.3.101:54321    ::ffff:10.20.3.104:58233    ESTABLISHED 5898/java
tcp        0      0 ::ffff:10.20.3.101:54321    ::ffff:10.20.3.104:58236    ESTABLISHED 5898/java
tcp        0      0 ::ffff:10.20.3.101:54321    ::ffff:10.20.3.104:58234    ESTABLISHED 5898/java
tcp        0      0 ::ffff:10.20.3.101:54321    ::ffff:10.20.3.104:58237    ESTABLISHED 5898/java
tcp   14390168      0 ::ffff:10.20.3.101:54321    ::ffff:10.20.3.104:58235    ESTABLISHED 5898/java

META FILEATTACHMENT attachment="transfer-notes-2012-11-01.png" attr="" comment="" date="1351841497" name="transfer-notes-2012-11-01.png" path="transfer-notes-2012-11-01.png" size="128899" user="igable" version="1"

Revision 52012-11-02 - igable

Line: 1 to 1
 

SuperComputing 2012

Line: 265 to 265
 Oct 29, 2012 10:48:19 AM lia.util.net.copy.FDTWriterSession doPostProcessing INFO: [ FDTWriterSession ] No post processing filters defined/processed.
Added:
>
>

Post firmware single clients

Halfway though the transfer the speed was cut in half.

01/11 20:16:34   Net In: 11.824 Gb/s   Avg: 16.476 Gb/s 99.90% ( 00s )
After the transfer of the files was complete FDT continued to report line like such:

01/11 20:16:30   Net Out: 11.686 Gb/s   Avg: 16.490 Gb/s 99.54% ( 03s )
01/11 20:16:35   Net Out: 11.770 Gb/s   Avg: 16.459 Gb/s 100.00% ( 00s )
01/11 20:16:40   Net Out: 0.000 b/s   Avg: 16.354 Gb/s 100.00% ( 00s )
01/11 20:16:45   Net Out: 0.000 b/s   Avg: 16.250 Gb/s 100.00% ( 00s )
01/11 20:16:50   Net Out: 0.000 b/s   Avg: 16.147 Gb/s 100.00% ( 00s )
01/11 20:16:55   Net Out: 0.000 b/s   Avg: 16.045 Gb/s 100.00% ( 00s )
01/11 20:17:00   Net Out: 0.000 b/s   Avg: 15.945 Gb/s 100.00% ( 00s )

until eventually killed.

The problem with half open connection was eliminated with the firmware upgrade:

tcp        0      1 ::ffff:10.20.3.104:54008    ::ffff:10.20.3.101:54321    SYN_SENT    5684/java
was fixed by moving to the 2.11.500 firmware version for the Mellanox CX3.

Using a single writer thread the transfer was completely stable.

01/11 22:33:59  Net In: 2.475 Gb/s      Avg: 10.954 Gb/s 100.00% ( 00s )

FDTWriterSession ( ff0ff011-80f1-4497-8397-72e1a18ab78b ) final stats:
 Started: Thu Nov 01 22:14:23 PDT 2012
 Ended:   Thu Nov 01 22:34:04 PDT 2012
 Transfer period:   19m 40s
 TotalBytes: 1609085802000
 TotalNetworkBytes: 1609085802000
 Exit Status: OK

Using 2 writer threads also completely stable:

01/11 23:04:14   Net In: 3.423 Gb/s   Avg: 21.328 Gb/s 100.00% ( 00s )

FDTWriterSession ( d9cad4f0-8987-4079-8ec0-7e8f73890afb ) final stats:
 Started: Thu Nov 01 22:54:09 PDT 2012
 Ended:   Thu Nov 01 23:04:18 PDT 2012
 Transfer period:   10m 09s
 TotalBytes: 1609085802000
 TotalNetworkBytes: 1609085802000
 Exit Status: OK

Revision 42012-10-29 - igable

Line: 1 to 1
 

SuperComputing 2012

Line: 126 to 126
  TotalNetworkBytes: 107374182400 Exit Status: OK
Added:
>
>

7 clients one server

9cee52db-3569-4b29-b894-06a536c14409Net In: 1.246 Gb/s  Avg: 1.216 Gb/s 99.14% ( 06s )
9ea1e5bc-9c3c-41c0-8772-0e9a8845cbd5Net In: 1.238 Gb/s  Avg: 1.219 Gb/s 99.38% ( 04s )
9f1983ee-585f-47be-9b76-5d6d6d51f688Net In: 1.226 Gb/s  Avg: 1.217 Gb/s 99.21% ( 05s )
b55d01b7-5d32-4b9c-a222-93d114a33bf7Net In: 1.201 Gb/s  Avg: 1.215 Gb/s 98.95% ( 07s )
13792c14-34e5-413e-9b2c-de1fded01a80Net In: 1.246 Gb/s  Avg: 1.225 Gb/s 99.85% ( 01s )
2456eb3d-1f19-45d8-9441-6213d1f60b39Net In: 1.209 Gb/s  Avg: 1.221 Gb/s 99.54% ( 03s )
5106e622-0934-4903-afc7-92b540759c85Net In: 1.201 Gb/s  Avg: 1.220 Gb/s 99.46% ( 03s )
Total Net In: 8.569 Gb/s


......




FDTWriterSession ( 13792c14-34e5-413e-9b2c-de1fded01a80 ) final stats:
 Started: Mon Oct 29 10:36:24 PDT 2012
 Ended:   Mon Oct 29 10:48:11 PDT 2012
 Transfer period:   11m 46s
 TotalBytes: 107374182400
 TotalNetworkBytes: 107374182400
 Exit Status: OK

Oct 29, 2012 10:48:11 AM lia.util.net.copy.FDTWriterSession doPostProcessing
INFO: [ FDTWriterSession ] Post Processing started
Oct 29, 2012 10:48:12 AM lia.util.net.copy.FDTWriterSession doPostProcessing
INFO: [ FDTWriterSession ] No post processing filters defined/processed.
Oct 29, 2012 10:48:12 AM lia.util.net.copy.transport.ControlChannel run
INFO:  ControlThread for ( 13792c14-34e5-413e-9b2c-de1fded01a80 ) /192.168.100.2:35549 FINISHED
Oct 29, 2012 10:48:13 AM lia.util.net.copy.FDTWriterSession handleEndFDTSession
INFO: [ FDTWriterSession ] Remote FDTReaderSession for session [ b55d01b7-5d32-4b9c-a222-93d114a33bf7 ] finished ok. Waiting for our side to finish.
Oct 29, 2012 10:48:14 AM lia.util.net.copy.transport.ControlChannel run
INFO:  ControlThread for ( 2456eb3d-1f19-45d8-9441-6213d1f60b39 ) /192.168.100.2:35552 FINISHED


FDTWriterSession ( 2456eb3d-1f19-45d8-9441-6213d1f60b39 ) final stats:
 Started: Mon Oct 29 10:36:24 PDT 2012
 Ended:   Mon Oct 29 10:48:14 PDT 2012
 Transfer period:   11m 49s
 TotalBytes: 107374182400
 TotalNetworkBytes: 107374182400
 Exit Status: OK

Oct 29, 2012 10:48:14 AM lia.util.net.copy.FDTWriterSession doPostProcessing
INFO: [ FDTWriterSession ] Post Processing started
Oct 29, 2012 10:48:14 AM lia.util.net.copy.FDTWriterSession doPostProcessing
INFO: [ FDTWriterSession ] No post processing filters defined/processed.


FDTWriterSession ( 5106e622-0934-4903-afc7-92b540759c85 ) final stats:
 Started: Mon Oct 29 10:36:24 PDT 2012
 Ended:   Mon Oct 29 10:48:15 PDT 2012
 Transfer period:   11m 50s
 TotalBytes: 107374182400
 TotalNetworkBytes: 107374182400
 Exit Status: OK

Oct 29, 2012 10:48:15 AM lia.util.net.copy.FDTWriterSession doPostProcessing
INFO: [ FDTWriterSession ] Post Processing started
Oct 29, 2012 10:48:15 AM lia.util.net.copy.FDTWriterSession doPostProcessing
INFO: [ FDTWriterSession ] No post processing filters defined/processed.
Oct 29, 2012 10:48:15 AM lia.util.net.copy.transport.ControlChannel run
INFO:  ControlThread for ( 5106e622-0934-4903-afc7-92b540759c85 ) /192.168.100.2:35553 FINISHED
29/10 10:48:15  7 active sessions:
9cee52db-3569-4b29-b894-06a536c14409Net In: 326.459 Mb/s        Avg: 1.209 Gb/s 100.00% ( 00s )
9ea1e5bc-9c3c-41c0-8772-0e9a8845cbd5Net In: 0.000 b/s   Avg: 1.209 Gb/s 100.00% ( 00s )
9f1983ee-585f-47be-9b76-5d6d6d51f688Net In: 254.261 Mb/s        Avg: 1.209 Gb/s 100.00% ( 00s )
b55d01b7-5d32-4b9c-a222-93d114a33bf7Net In: 681.169 Mb/s        Avg: 1.210 Gb/s 100.00% ( 00s )
Total Net In: 1.262 Gb/s


FDTWriterSession ( 9ea1e5bc-9c3c-41c0-8772-0e9a8845cbd5 ) final stats:
 Started: Mon Oct 29 10:36:25 PDT 2012
 Ended:   Mon Oct 29 10:48:16 PDT 2012
 Transfer period:   11m 51s
 TotalBytes: 107374182400
 TotalNetworkBytes: 107374182400
 Exit Status: OK

Oct 29, 2012 10:48:16 AM lia.util.net.copy.FDTWriterSession doPostProcessing
INFO: [ FDTWriterSession ] Post Processing started
Oct 29, 2012 10:48:16 AM lia.util.net.copy.FDTWriterSession doPostProcessing
INFO: [ FDTWriterSession ] No post processing filters defined/processed.
Oct 29, 2012 10:48:16 AM lia.util.net.copy.transport.ControlChannel run
INFO:  ControlThread for ( 9ea1e5bc-9c3c-41c0-8772-0e9a8845cbd5 ) /192.168.100.2:35554 FINISHED
Oct 29, 2012 10:48:18 AM lia.util.net.copy.transport.ControlChannel run
INFO:  ControlThread for ( 9f1983ee-585f-47be-9b76-5d6d6d51f688 ) /192.168.100.2:35551 FINISHED
Oct 29, 2012 10:48:18 AM lia.util.net.copy.transport.ControlChannel run
INFO:  ControlThread for ( 9cee52db-3569-4b29-b894-06a536c14409 ) /192.168.100.2:35550 FINISHED


FDTWriterSession ( 9f1983ee-585f-47be-9b76-5d6d6d51f688 ) final stats:
 Started: Mon Oct 29 10:36:24 PDT 2012
 Ended:   Mon Oct 29 10:48:18 PDT 2012
 Transfer period:   11m 53s
 TotalBytes: 107374182400
 TotalNetworkBytes: 107374182400
 Exit Status: OK

Oct 29, 2012 10:48:18 AM lia.util.net.copy.FDTWriterSession doPostProcessing
INFO: [ FDTWriterSession ] Post Processing started
Oct 29, 2012 10:48:18 AM lia.util.net.copy.FDTWriterSession doPostProcessing
INFO: [ FDTWriterSession ] No post processing filters defined/processed.


FDTWriterSession ( 9cee52db-3569-4b29-b894-06a536c14409 ) final stats:
 Started: Mon Oct 29 10:36:24 PDT 2012
 Ended:   Mon Oct 29 10:48:18 PDT 2012
 Transfer period:   11m 53s
 TotalBytes: 107374182400
 TotalNetworkBytes: 107374182400
 Exit Status: OK

Oct 29, 2012 10:48:18 AM lia.util.net.copy.FDTWriterSession doPostProcessing
INFO: [ FDTWriterSession ] Post Processing started
Oct 29, 2012 10:48:18 AM lia.util.net.copy.FDTWriterSession doPostProcessing
INFO: [ FDTWriterSession ] No post processing filters defined/processed.


FDTWriterSession ( b55d01b7-5d32-4b9c-a222-93d114a33bf7 ) final stats:
 Started: Mon Oct 29 10:36:25 PDT 2012
 Ended:   Mon Oct 29 10:48:19 PDT 2012
 Transfer period:   11m 54s
 TotalBytes: 107374182400
 TotalNetworkBytes: 107374182400
 Exit Status: OK

Oct 29, 2012 10:48:19 AM lia.util.net.copy.transport.ControlChannel run
INFO:  ControlThread for ( b55d01b7-5d32-4b9c-a222-93d114a33bf7 ) /192.168.100.2:35548 FINISHED
Oct 29, 2012 10:48:19 AM lia.util.net.copy.FDTWriterSession doPostProcessing
INFO: [ FDTWriterSession ] Post Processing started
Oct 29, 2012 10:48:19 AM lia.util.net.copy.FDTWriterSession doPostProcessing
INFO: [ FDTWriterSession ] No post processing filters defined/processed.

Revision 32012-10-29 - igable

Line: 1 to 1
 

SuperComputing 2012

Added:
>
>

Memory to Memory

No problems with memoty to memory: 37.806 Gb/s

[root@sc02 ~]# fdtClient -P 7 -c 192.168.100.4  /dev/zero -d /dev/null
Oct 28, 2012 11:49:59 PM lia.util.net.common.Config <init>
INFO: Using lia.util.net.copy.PosixFSFileChannelProviderFactory as FileChannelProviderFactory
Oct 28, 2012 11:49:59 PM lia.util.net.common.Config <init>
INFO: FDT started in client mode
FDT uses *blocking* I/O mode.
INFO: Requested window size -1. Using window size: 49360
28/10 23:50:09   Net Out: 38.130 Gb/s   Avg: 38.130 Gb/s
28/10 23:50:14   Net Out: 38.128 Gb/s   Avg: 38.129 Gb/s
28/10 23:50:19   Net Out: 36.775 Gb/s   Avg: 37.678 Gb/s
28/10 23:50:24   Net Out: 38.197 Gb/s   Avg: 37.806 Gb/s
 

Disk to Disk within a machine

MegaCli -CfgLdAdd -r0 [252:1] WT NORA DIRECT -strpsz 1024 -a0
Line: 8 to 28
 [Sun Oct 28 20:56:57 PDT 2012] Current Speed = 441.648 MB/s Avg Speed: 436.479 MB/s Total Transfer: 4.277 GB
Changed:
<
<

Network transfer with 7 seperate Virtual disks

>
>

Network transfer with 7 virtual disks and 1 fdtServers, 7 parallel streams

Grand total with 7 disks, 1 FDT Server: 10.533 Gb/s

[root@sc02 ssd6]# fdtClient -P 7 -c 192.168.100.4 -fl /ssd6/filelist.txt -d /
Avg: 10.533 Gb/s 100.00% ( 00s )
FDTReaderSession ( 13f62162-4fb5-41c5-b4fb-db9af9c453f3 ) final stats:
 Started: Sun Oct 28 23:33:07 PDT 2012
 Ended:   Sun Oct 28 23:42:42 PDT 2012
 Transfer period:   09m 34s
 TotalBytes: 751619276800
 TotalNetworkBytes: 751619276800
 Exit Status: OK

Network transfer with 7 virtual disks and 7 fdtServers

The test was set up with 1 virtual disk per physical disk and 1 FDT server for every disk.

Grand total with 7 disks, 7 FDT Servers: 22.196 Gb/s

fdtClient -c 192.168.100.4 -p 54321 -d /ssd1/  /ssd1/100Gfile_x.dat > /root/fdt1.log &
fdtClient -c 192.168.100.4 -p 54322 -d /ssd2/  /ssd2/100Gfile_x.dat > /root/fdt2.log &
fdtClient -c 192.168.100.4 -p 54323 -d /ssd3/  /ssd3/100Gfile_x.dat > /root/fdt3.log &
fdtClient -c 192.168.100.4 -p 54324 -d /ssd4/  /ssd4/100Gfile_x.dat > /root/fdt4.log &
fdtClient -c 192.168.100.4 -p 54325 -d /ssd5/  /ssd5/100Gfile_x.dat > /root/fdt5.log &
fdtClient -c 192.168.100.4 -p 54326 -d /ssd6/  /ssd6/100Gfile_x.dat > /root/fdt6.log &
fdtClient -c 192.168.100.4 -p 54327 -d /ssd7/  /ssd7/100Gfile_x.dat > /root/fdt7.log &
 
Changed:
<
<
fdtClient -P 7 -c 192.168.100.4 -fl /ssd7/filelist.txt -d / <28/10 21:44:58 Net In: 12.912 Gb/s Avg: 12.912 Gb/s 28/10 21:45:03 Net In: 12.062 Gb/s Avg: 12.487 Gb/s 28/10 21:45:08 Net In: 11.944 Gb/s Avg: 12.305 Gb/s 28/10 21:45:13 Net In: 11.654 Gb/s Avg: 12.142 Gb/s 42.00% ( 28s ) 28/10 21:45:18 Net In: 9.740 Gb/s Avg: 11.661 Gb/s 50.10% ( 25s ) 28/10 21:45:23 Net In: 9.751 Gb/s Avg: 11.343 Gb/s 58.21% ( 22s ) 28/10 21:45:28 Net In: 9.956 Gb/s Avg: 11.145 Gb/s 66.49% ( 18s ) 28/10 21:45:33 Net In: 9.199 Gb/s Avg: 10.901 Gb/s 74.13% ( 14s ) 28/10 21:45:38 Net In: 9.302 Gb/s Avg: 10.724 Gb/s 81.87% ( 10s ) 28/10 21:45:43 Net In: 6.399 Gb/s Avg: 10.291 Gb/s 87.19% ( 07s ) 28/10 21:45:48 Net In: 7.503 Gb/s Avg: 10.038 Gb/s 93.43% ( 03s ) 28/10 21:45:53 Net In: 6.156 Gb/s Avg: 9.714 Gb/s 98.55% ( 00s ) Oct 28, 2012 9:45:56 PM lia.util.net.copy.FDTWriterSession handleEndFDTSession INFO: [ FDTWriterSession ] Remote FDTReaderSession for session [ 496370fc-e428-4985-a588-8856e8c03774 ] finished ok. Waiting for our side to finish. 28/10 21:45:58 Net In: 1.746 Gb/s Avg: 9.101 Gb/s 100.00% ( 00s )
>
>
Avg: 3.067 Gb/s 100.00% ( 00s ) FDTWriterSession ( 9920c20c-6378-4ef1-b14b-8393b5c1eafb ) final stats: Started: Sun Oct 28 22:57:04 PDT 2012 Ended: Sun Oct 28 23:01:48 PDT 2012 Transfer period: 04m 43s TotalBytes: 107374182400 TotalNetworkBytes: 107374182400 Exit Status: OK

Avg: 2.962 Gb/s 100.00% ( 00s ) FDTWriterSession ( 2cc841b1-0ce5-4f09-b773-794e59d0d35e ) final stats: Started: Sun Oct 28 22:57:04 PDT 2012 Ended: Sun Oct 28 23:01:58 PDT 2012 Transfer period: 04m 53s TotalBytes: 107374182400 TotalNetworkBytes: 107374182400 Exit Status: OK

Avg: 3.159 Gb/s 100.00% ( 00s ) FDTWriterSession ( 183111ea-8c54-4247-870f-e1f7eb8440b7 ) final stats: Started: Sun Oct 28 22:57:04 PDT 2012 Ended: Sun Oct 28 23:01:42 PDT 2012 Transfer period: 04m 37s TotalBytes: 107374182400 TotalNetworkBytes: 107374182400 Exit Status: OK

Avg: 3.232 Gb/s 100.00% ( 00s ) FDTWriterSession ( 9bf985f6-9260-466d-b775-99f4df5931a6 ) final stats: Started: Sun Oct 28 22:57:04 PDT 2012 Ended: Sun Oct 28 23:01:33 PDT 2012 Transfer period: 04m 28s TotalBytes: 107374182400 TotalNetworkBytes: 107374182400 Exit Status: OK

Avg: 3.297 Gb/s 100.00% ( 00s ) FDTWriterSession ( 45f42c87-44c8-4db8-8d29-fcc2b33a12a1 ) final stats: Started: Sun Oct 28 22:57:04 PDT 2012 Ended: Sun Oct 28 23:01:31 PDT 2012 Transfer period: 04m 26s TotalBytes: 107374182400 TotalNetworkBytes: 107374182400 Exit Status: OK

Avg: 3.238 Gb/s 100.00% ( 00s ) FDTWriterSession ( 5dedbe6a-949b-47ec-9d8a-2e9c0b32e5f3 ) final stats: Started: Sun Oct 28 22:57:04 PDT 2012 Ended: Sun Oct 28 23:01:36 PDT 2012 Transfer period: 04m 31s TotalBytes: 107374182400 TotalNetworkBytes: 107374182400 Exit Status: OK

Avg: 3.241 Gb/s 100.00% ( 00s ) FDTWriterSession ( 90247407-4dfd-417b-8022-bc3925e0f078 ) final stats: Started: Sun Oct 28 22:57:04 PDT 2012 Ended: Sun Oct 28 23:01:35 PDT 2012 Transfer period: 04m 30s TotalBytes: 107374182400 TotalNetworkBytes: 107374182400 Exit Status: OK

 

Revision 22012-10-29 - igable

Line: 1 to 1
 

SuperComputing 2012

Changed:
<
<

Vertex 4 Benchmarks

Write to disk from /dev/zero

  • First attempt no tunning. Only SATA 2 slot.
>
>

Disk to Disk within a machine

 
Changed:
<
<
[root@elephant09 ~]# java -cp /root/fdt/fdt.jar lia.util.net.common.DDCopy if=/dev/zero of=/vertex/10Goutputfile5 bs=10M count=10240 [Fri Aug 24 16:46:16 PDT 2012] Current Speed = 1.25 GB/s Avg Speed: 1.25 GB/s Total Transfer: 2.5 GB [Fri Aug 24 16:46:18 PDT 2012] Current Speed = 610.236 MB/s Avg Speed: 942.46 MB/s Total Transfer: 3.711 GB [Fri Aug 24 16:46:20 PDT 2012] Current Speed = 129.935 MB/s Avg Speed: 672.965 MB/s Total Transfer: 3.965 GB [Fri Aug 24 16:46:22 PDT 2012] Current Speed = 129.935 MB/s Avg Speed: 537.715 MB/s Total Transfer: 4.219 GB

[Fri Aug 24 16:58:48 PDT 2012] Current Speed = 130 MB/s Avg Speed: 135.016 MB/s Total Transfer: 99.453 GB [Fri Aug 24 16:58:50 PDT 2012] Current Speed = 129.935 MB/s Avg Speed: 135.003 MB/s Total Transfer: 99.707 GB [Fri Aug 24 16:58:52 PDT 2012] Current Speed = 130 MB/s Avg Speed: 134.99 MB/s Total Transfer: 99.961 GB

Total Transfer: 100 GBytes ( 107374182400 bytes ) Time: 758 seconds Avg Speed: 134.993 MB/s

>
>
MegaCli -CfgLdAdd -r0 [252:1] WT NORA DIRECT -strpsz 1024 -a0 [root@sc02 ~]# fdtCopy if=/ssd1/010Gfile_n0010.dat of=/ssd7/010Gfile_n0010.dat [Sun Oct 28 20:56:57 PDT 2012] Current Speed = 441.648 MB/s Avg Speed: 436.479 MB/s Total Transfer: 4.277 GB
 
Changed:
<
<

Read from disk to /dev/null

  • note that the disk was unmounted in between the write and the read.
[root@elephant09 ~]# java -cp fdt.jar lia.util.net.common.DDCopy if=/vertex/10Goutputfile5 of=/dev/null
Error: Could not find or load main class lia.util.net.common.DDCopy
[root@elephant09 ~]# java -cp /root/fdt/fdt.jar lia.util.net.common.DDCopy if=/vertex/10Goutputfile5 of=/dev/null
[Fri Aug 24 17:03:26 PDT 2012] Current Speed = 133.93 MB/s Avg Speed: 133.93 MB/s Total Transfer: 267.859 MB
[Fri Aug 24 17:03:28 PDT 2012] Current Speed = 135.468 MB/s Avg Speed: 134.705 MB/s Total Transfer: 542.859 MB


[Fri Aug 24 17:15:58 PDT 2012] Current Speed = 135.5 MB/s Avg Speed: 135.46 MB/s Total Transfer: 99.775 GB
>
>

Network transfer with 7 seperate Virtual disks

 
Changed:
<
<
Total Transfer: 100 GBytes ( 107374182400 bytes ) Time: 755 seconds Avg Speed: 135.46 MB/s
>
>
fdtClient -P 7 -c 192.168.100.4 -fl /ssd7/filelist.txt -d /
<28/10 21:44:58   Net In: 12.912 Gb/s   Avg: 12.912 Gb/s
28/10 21:45:03   Net In: 12.062 Gb/s   Avg: 12.487 Gb/s
28/10 21:45:08   Net In: 11.944 Gb/s   Avg: 12.305 Gb/s
28/10 21:45:13   Net In: 11.654 Gb/s   Avg: 12.142 Gb/s 42.00% ( 28s )
28/10 21:45:18   Net In: 9.740 Gb/s   Avg: 11.661 Gb/s 50.10% ( 25s )
28/10 21:45:23   Net In: 9.751 Gb/s   Avg: 11.343 Gb/s 58.21% ( 22s )
28/10 21:45:28   Net In: 9.956 Gb/s   Avg: 11.145 Gb/s 66.49% ( 18s )
28/10 21:45:33   Net In: 9.199 Gb/s   Avg: 10.901 Gb/s 74.13% ( 14s )
28/10 21:45:38   Net In: 9.302 Gb/s   Avg: 10.724 Gb/s 81.87% ( 10s )
28/10 21:45:43   Net In: 6.399 Gb/s   Avg: 10.291 Gb/s 87.19% ( 07s )
28/10 21:45:48   Net In: 7.503 Gb/s   Avg: 10.038 Gb/s 93.43% ( 03s )
28/10 21:45:53   Net In: 6.156 Gb/s   Avg: 9.714 Gb/s 98.55% ( 00s )
Oct 28, 2012 9:45:56 PM lia.util.net.copy.FDTWriterSession handleEndFDTSession
INFO: [ FDTWriterSession ] Remote FDTReaderSession for session [ 496370fc-e428-4985-a588-8856e8c03774 ] finished ok. Waiting for our side to finish.
28/10 21:45:58   Net In: 1.746 Gb/s   Avg: 9.101 Gb/s 100.00% ( 00s )
 

Revision 12012-08-25 - igable

Line: 1 to 1
Added:
>
>

SuperComputing 2012

Vertex 4 Benchmarks

Write to disk from /dev/zero

  • First attempt no tunning. Only SATA 2 slot.

[root@elephant09 ~]# java -cp /root/fdt/fdt.jar lia.util.net.common.DDCopy if=/dev/zero of=/vertex/10Goutputfile5 bs=10M count=10240
[Fri Aug 24 16:46:16 PDT 2012] Current Speed = 1.25 GB/s Avg Speed: 1.25 GB/s Total Transfer: 2.5 GB
[Fri Aug 24 16:46:18 PDT 2012] Current Speed = 610.236 MB/s Avg Speed: 942.46 MB/s Total Transfer: 3.711 GB
[Fri Aug 24 16:46:20 PDT 2012] Current Speed = 129.935 MB/s Avg Speed: 672.965 MB/s Total Transfer: 3.965 GB
[Fri Aug 24 16:46:22 PDT 2012] Current Speed = 129.935 MB/s Avg Speed: 537.715 MB/s Total Transfer: 4.219 GB

[Fri Aug 24 16:58:48 PDT 2012] Current Speed = 130 MB/s Avg Speed: 135.016 MB/s Total Transfer: 99.453 GB
[Fri Aug 24 16:58:50 PDT 2012] Current Speed = 129.935 MB/s Avg Speed: 135.003 MB/s Total Transfer: 99.707 GB
[Fri Aug 24 16:58:52 PDT 2012] Current Speed = 130 MB/s Avg Speed: 134.99 MB/s Total Transfer: 99.961 GB


 Total Transfer: 100 GBytes ( 107374182400 bytes )
 Time: 758 seconds
 Avg Speed: 134.993 MB/s

Read from disk to /dev/null

  • note that the disk was unmounted in between the write and the read.
[root@elephant09 ~]# java -cp fdt.jar lia.util.net.common.DDCopy if=/vertex/10Goutputfile5 of=/dev/null
Error: Could not find or load main class lia.util.net.common.DDCopy
[root@elephant09 ~]# java -cp /root/fdt/fdt.jar lia.util.net.common.DDCopy if=/vertex/10Goutputfile5 of=/dev/null
[Fri Aug 24 17:03:26 PDT 2012] Current Speed = 133.93 MB/s Avg Speed: 133.93 MB/s Total Transfer: 267.859 MB
[Fri Aug 24 17:03:28 PDT 2012] Current Speed = 135.468 MB/s Avg Speed: 134.705 MB/s Total Transfer: 542.859 MB


[Fri Aug 24 17:15:58 PDT 2012] Current Speed = 135.5 MB/s Avg Speed: 135.46 MB/s Total Transfer: 99.775 GB


 Total Transfer: 100 GBytes ( 107374182400 bytes )
 Time: 755 seconds
 Avg Speed: 135.46 MB/s
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback