100G Notes

Diagrams

I/0 Summary

Test TypeSorted descending 5 x Deneva 2 Raid 0 Vertex 3
Write to disk from /dev/zero (disk only) 5.131 Gb/s 3.081 Gb/s *
SSD to Memory with FDT (network) 6.580 Gb/s 2.970 Gb/s
Read from Disk to /dev/null (disk only) 6.105 Gb/s 2.752 Gb/s

* higher bandwidth the the SATA 2 bus hence a suspect number

Final Config

Configuration Dell R710:

  • PERC H700 Internal
  • 12 GB Memory 1333 MHz
  • 5 x 233 GB Deneva SSD RAID-0 1 MB stripe
  • 2 x 146 GB 10k SAS RAID 1 System Disk
  • Intel X520 DA

Test Result
Memory To Memory 9.9 Gb/s
Disk to Memory 9.9 Gb/s
Disk to Disk 8.8 Gb/s

SSD to Memory with FDT 

root@scdemo01 ~]# umount /ssd/
[root@scdemo01 ~]# mount /ssd
[root@scdemo01 ~]# java -jar /root/fdt/fdt.jar -c uvic-sonar.hepnetcanada.ca -d /dev/null -fl /ssd/filelist.txt


FDT [ 0.9.23-201107290935 ] STARTED ... 


Oct 21, 2011 5:26:01 AM lia.util.net.common.Config <init>
INFO: Using lia.util.net.copy.PosixFSFileChannelProviderFactory as FileChannelProviderFactory
Oct 21, 2011 5:26:01 AM lia.util.net.common.Config <init>
INFO: FDT started in client mode
FDT uses *blocking* I/O mode.
Oct 21, 2011 5:26:02 AM lia.util.net.copy.transport.ControlChannel initStreams
INFO: NEW CONTROL stream for 1e0d3078-5e84-40e8-8a05-2864160577cb initialized 
Oct 21, 2011 5:26:02 AM lia.util.net.copy.transport.ControlChannel$ControlChannelPingerTask <init>
INFO: [ ControlChannelPingerTask ] initialized
Oct 21, 2011 5:26:02 AM lia.util.net.copy.FDTReaderSession internalInit
INFO: [ FDTReaderSession ] No pre processing filters defined/processed.
Oct 21, 2011 5:26:03 AM lia.util.net.copy.FDTReaderSession startReading
INFO:  Started DiskReaderTasks for the following partions [ 64769 ] for FDTSession: 1e0d3078-5e84-40e8-8a05-2864160577cb
Oct 21, 2011 5:26:03 AM lia.util.net.copy.transport.TCPTransportProvider tryToConnect
INFO: Requested window size -1. Using window size: 43690
21/10 05:26:11   Net Out: 9.879 Gb/s   Avg: 9.879 Gb/s
21/10 05:26:17   Net Out: 9.904 Gb/s   Avg: 9.891 Gb/s
21/10 05:26:22   Net Out: 9.889 Gb/s   Avg: 9.890 Gb/s
21/10 05:26:27   Net Out: 9.897 Gb/s   Avg: 9.892 Gb/s 68.87% ( 10s )
21/10 05:26:32   Net Out: 9.884 Gb/s   Avg: 9.890 Gb/s 83.25% ( 05s )
21/10 05:26:37   Net Out: 9.899 Gb/s   Avg: 9.891 Gb/s 97.66% ( 00s )
Oct 21, 2011 5:26:38 AM lia.util.net.copy.FDTReaderSession handleEndFDTSession
INFO: [ FDTReaderSession ] Remote FDTWriterSession for session [ 1e0d3078-5e84-40e8-8a05-2864160577cb ] finished OK!


FDTReaderSession ( 1e0d3078-5e84-40e8-8a05-2864160577cb ) final stats:
 Started: Fri Oct 21 05:26:02 PDT 2011
 Ended:   Fri Oct 21 05:26:41 PDT 2011
 Transfer period:   39s
 TotalBytes: 42949672960
 TotalNetworkBytes: 42949672960
 Exit Status: OK

Oct 21, 2011 5:26:41 AM lia.util.net.copy.FDTReaderSession doPostProcessing
INFO: [ FDTReaderSession ] Post Processing started
Oct 21, 2011 5:26:41 AM lia.util.net.copy.FDTReaderSession doPostProcessing
INFO: [ FDTReaderSession ] No post processing filters defined/processed.
 [ Fri Oct 21 05:26:42 PDT 2011 ] - GracefulStopper hook started ... Waiting for the cleanup to finish
 [ Fri Oct 21 05:26:42 PDT 2011 ]  - GracefulStopper hook finished!

 [ Fri Oct 21 05:26:42 PDT 2011 ]  FDT Session finished OK.

Disk to Disk with FDT 

[root@scdemo01 ~]# java -jar /root/fdt/fdt.jar -c elephant71.heprc.uvic.ca -d /ssd -fl /ssd/filelist.txt


FDT [ 0.9.23-201107290935 ] STARTED ... 


Oct 21, 2011 5:47:13 AM lia.util.net.common.Config <init>
INFO: Using lia.util.net.copy.PosixFSFileChannelProviderFactory as FileChannelProviderFactory
Oct 21, 2011 5:47:13 AM lia.util.net.common.Config <init>
INFO: FDT started in client mode
FDT uses *blocking* I/O mode.
Oct 21, 2011 5:47:14 AM lia.util.net.copy.transport.ControlChannel initStreams
INFO: NEW CONTROL stream for f40d03b8-0dae-4c6a-b117-ca75d1d78f7f initialized 
Oct 21, 2011 5:47:14 AM lia.util.net.copy.transport.ControlChannel$ControlChannelPingerTask <init>
INFO: [ ControlChannelPingerTask ] initialized
Oct 21, 2011 5:47:14 AM lia.util.net.copy.FDTReaderSession internalInit
INFO: [ FDTReaderSession ] No pre processing filters defined/processed.
Oct 21, 2011 5:47:15 AM lia.util.net.copy.FDTReaderSession startReading
INFO:  Started DiskReaderTasks for the following partions [ 64769 ] for FDTSession: f40d03b8-0dae-4c6a-b117-ca75d1d78f7f
Oct 21, 2011 5:47:15 AM lia.util.net.copy.transport.TCPTransportProvider tryToConnect
INFO: Requested window size -1. Using window size: 43690
21/10 05:47:23   Net Out: 9.872 Gb/s   Avg: 9.872 Gb/s
21/10 05:47:28   Net Out: 8.413 Gb/s   Avg: 9.142 Gb/s
21/10 05:47:33   Net Out: 8.946 Gb/s   Avg: 9.076 Gb/s
21/10 05:47:38   Net Out: 9.215 Gb/s   Avg: 9.111 Gb/s 63.84% ( 13s )
21/10 05:47:43   Net Out: 8.272 Gb/s   Avg: 8.943 Gb/s 75.88% ( 09s )
21/10 05:47:48   Net Out: 10.020 Gb/s   Avg: 9.122 Gb/s 90.46% ( 03s )
21/10 05:47:53   Net Out: 6.559 Gb/s   Avg: 8.756 Gb/s 100.00% ( 00s )
Oct 21, 2011 5:47:54 AM lia.util.net.copy.FDTReaderSession handleEndFDTSession
INFO: [ FDTReaderSession ] Remote FDTWriterSession for session [ f40d03b8-0dae-4c6a-b117-ca75d1d78f7f ] finished OK!


FDTReaderSession ( f40d03b8-0dae-4c6a-b117-ca75d1d78f7f ) final stats:
 Started: Fri Oct 21 05:47:13 PDT 2011
 Ended:   Fri Oct 21 05:47:57 PDT 2011
 Transfer period:   43s
 TotalBytes: 42949672960
 TotalNetworkBytes: 42949672960
 Exit Status: OK

Oct 21, 2011 5:47:57 AM lia.util.net.copy.FDTReaderSession doPostProcessing
INFO: [ FDTReaderSession ] Post Processing started
Oct 21, 2011 5:47:57 AM lia.util.net.copy.FDTReaderSession doPostProcessing
INFO: [ FDTReaderSession ] No post processing filters defined/processed.
 [ Fri Oct 21 05:47:58 PDT 2011 ] - GracefulStopper hook started ... Waiting for the cleanup to finish
 [ Fri Oct 21 05:47:58 PDT 2011 ]  - GracefulStopper hook finished!

 [ Fri Oct 21 05:47:58 PDT 2011 ]  FDT Session finished OK.

Memory to Memory with FDT 

[root@scdemo01 fdt]# java -jar fdt.jar -c elephant71.heprc.uvic.ca -bio -d /dev/null /dev/zero


FDT [ 0.9.23-201107290935 ] STARTED ... 


Oct 21, 2011 6:45:14 AM lia.util.net.common.Config <init>
INFO: Using lia.util.net.copy.PosixFSFileChannelProviderFactory as FileChannelProviderFactory
Oct 21, 2011 6:45:14 AM lia.util.net.common.Config <init>
INFO: FDT started in client mode
FDT uses *blocking* I/O mode.
Oct 21, 2011 6:45:14 AM lia.util.net.copy.transport.ControlChannel initStreams
INFO: NEW CONTROL stream for 810f60b9-6341-4a68-964a-36225e7c7391 initialized 
Oct 21, 2011 6:45:14 AM lia.util.net.copy.transport.ControlChannel$ControlChannelPingerTask <init>
INFO: [ ControlChannelPingerTask ] initialized
Oct 21, 2011 6:45:14 AM lia.util.net.copy.FDTReaderSession internalInit
INFO: [ FDTReaderSession ] No pre processing filters defined/processed.
Oct 21, 2011 6:45:15 AM lia.util.net.copy.FDTReaderSession startReading
INFO:  Started DiskReaderTasks for the following partions [ 0 ] for FDTSession: 810f60b9-6341-4a68-964a-36225e7c7391
Oct 21, 2011 6:45:15 AM lia.util.net.copy.transport.TCPTransportProvider tryToConnect
INFO: Requested window size -1. Using window size: 43690
21/10 06:45:24   Net Out: 9.948 Gb/s   Avg: 9.948 Gb/s
21/10 06:45:29   Net Out: 9.832 Gb/s   Avg: 9.890 Gb/s
21/10 06:45:34   Net Out: 9.902 Gb/s   Avg: 9.893 Gb/s
21/10 06:45:39   Net Out: 9.978 Gb/s   Avg: 9.915 Gb/s
21/10 06:45:44   Net Out: 9.926 Gb/s   Avg: 9.916 Gb/s
21/10 06:45:49   Net Out: 9.852 Gb/s   Avg: 9.906 Gb/s

Memory to Disk with FDT 

[root@scdemo01 fdt]# java -jar fdt.jar -c elephant71.heprc.uvic.ca -bio -d /ssd/junk /dev/zero


FDT [ 0.9.23-201107290935 ] STARTED ... 


Oct 21, 2011 6:48:33 AM lia.util.net.common.Config <init>
INFO: Using lia.util.net.copy.PosixFSFileChannelProviderFactory as FileChannelProviderFactory
Oct 21, 2011 6:48:33 AM lia.util.net.common.Config <init>
INFO: FDT started in client mode
FDT uses *blocking* I/O mode.
Oct 21, 2011 6:48:33 AM lia.util.net.copy.transport.ControlChannel initStreams
INFO: NEW CONTROL stream for ef825917-99d0-4cd0-9ee8-323e4b7707b0 initialized 
Oct 21, 2011 6:48:33 AM lia.util.net.copy.transport.ControlChannel$ControlChannelPingerTask <init>
INFO: [ ControlChannelPingerTask ] initialized
Oct 21, 2011 6:48:33 AM lia.util.net.copy.FDTReaderSession internalInit
INFO: [ FDTReaderSession ] No pre processing filters defined/processed.
Oct 21, 2011 6:48:34 AM lia.util.net.copy.FDTReaderSession startReading
INFO:  Started DiskReaderTasks for the following partions [ 0 ] for FDTSession: ef825917-99d0-4cd0-9ee8-323e4b7707b0
Oct 21, 2011 6:48:34 AM lia.util.net.copy.transport.TCPTransportProvider tryToConnect
INFO: Requested window size -1. Using window size: 43690
21/10 06:48:43   Net Out: 9.931 Gb/s   Avg: 9.931 Gb/s
21/10 06:48:48   Net Out: 9.976 Gb/s   Avg: 9.954 Gb/s
21/10 06:48:53   Net Out: 9.921 Gb/s   Avg: 9.942 Gb/s
21/10 06:48:58   Net Out: 9.913 Gb/s   Avg: 9.935 Gb/s
21/10 06:49:03   Net Out: 9.968 Gb/s   Avg: 9.941 Gb/s
21/10 06:49:08   Net Out: 10.052 Gb/s   Avg: 9.959 Gb/s
21/10 06:49:13   Net Out: 9.834 Gb/s   Avg: 9.941 Gb/s
21/10 06:49:18   Net Out: 10.060 Gb/s   Avg: 9.956 Gb/s
21/10 06:49:23   Net Out: 9.795 Gb/s   Avg: 9.938 Gb/s

Memory to Disk with FDT

Not sure if this is Valid Test 

[root@scdemo01 fdt]# java -jar fdt.jar -c elephant71.heprc.uvic.ca -bio -d /ssd/junk /dev/zero


FDT [ 0.9.23-201107290935 ] STARTED ... 


Oct 21, 2011 6:51:32 AM lia.util.net.common.Config <init>
INFO: Using lia.util.net.copy.PosixFSFileChannelProviderFactory as FileChannelProviderFactory
Oct 21, 2011 6:51:32 AM lia.util.net.common.Config <init>
INFO: FDT started in client mode
FDT uses *blocking* I/O mode.
Oct 21, 2011 6:51:32 AM lia.util.net.copy.transport.ControlChannel initStreams
INFO: NEW CONTROL stream for abeb5348-c93c-4e39-8080-fac44a816a47 initialized 
Oct 21, 2011 6:51:32 AM lia.util.net.copy.transport.ControlChannel$ControlChannelPingerTask <init>
INFO: [ ControlChannelPingerTask ] initialized
Oct 21, 2011 6:51:32 AM lia.util.net.copy.FDTReaderSession internalInit
INFO: [ FDTReaderSession ] No pre processing filters defined/processed.
Oct 21, 2011 6:51:33 AM lia.util.net.copy.FDTReaderSession startReading
INFO:  Started DiskReaderTasks for the following partions [ 0 ] for FDTSession: abeb5348-c93c-4e39-8080-fac44a816a47
Oct 21, 2011 6:51:33 AM lia.util.net.copy.transport.TCPTransportProvider tryToConnect
INFO: Requested window size -1. Using window size: 43690
21/10 06:51:42   Net Out: 9.946 Gb/s   Avg: 9.946 Gb/s
21/10 06:51:47   Net Out: 9.891 Gb/s   Avg: 9.917 Gb/s
21/10 06:51:52   Net Out: 9.919 Gb/s   Avg: 9.918 Gb/s
21/10 06:51:57   Net Out: 10.003 Gb/s   Avg: 9.939 Gb/s
21/10 06:52:02   Net Out: 10.020 Gb/s   Avg: 9.955 Gb/s
21/10 06:52:07   Net Out: 9.887 Gb/s   Avg: 9.944 Gb/s
21/10 06:52:12   Net Out: 9.830 Gb/s   Avg: 9.927 Gb/s
21/10 06:52:17   Net Out: 9.956 Gb/s   Avg: 9.931 Gb/s
21/10 06:52:22   Net Out: 9.924 Gb/s   Avg: 9.930 Gb/s
21/10 06:52:27   Net Out: 9.899 Gb/s   Avg: 9.927 Gb/s
21/10 06:52:32   Net Out: 10.011 Gb/s   Avg: 9.935 Gb/s
21/10 06:52:37   Net Out: 9.847 Gb/s   Avg: 9.927 Gb/s
21/10 06:52:42   Net Out: 9.918 Gb/s   Avg: 9.926 Gb/s
21/10 06:52:47   Net Out: 9.919 Gb/s   Avg: 9.926 Gb/s
21/10 06:52:52   Net Out: 9.936 Gb/s   Avg: 9.926 Gb/s
21/10 06:52:57   Net Out: 9.869 Gb/s   Avg: 9.923 Gb/s

OCZ Deneva 2 Network Test

SSD to Memory with FDT 

[root@localhost deneva]# java -jar /root/fdt/fdt.jar -c uvic-sonar.hepnetcanada.ca -bio -d /dev/null 10Gfile.dat 10Gfile2.dat 10Gfile3.dat 


FDT [ 0.9.23-201107290935 ] STARTED ... 


Oct 17, 2011 12:30:08 AM lia.util.net.common.Config <init>
INFO: Using lia.util.net.copy.PosixFSFileChannelProviderFactory as FileChannelProviderFactory
Oct 17, 2011 12:30:08 AM lia.util.net.common.Config <init>
INFO: FDT started in client mode
FDT uses *blocking* I/O mode.
Oct 17, 2011 12:30:08 AM lia.util.net.copy.transport.ControlChannel initStreams
INFO: NEW CONTROL stream for 8d745066-e6f9-4fad-9446-eed7139f6221 initialized 
Oct 17, 2011 12:30:08 AM lia.util.net.copy.transport.ControlChannel$ControlChannelPingerTask <init>
INFO: [ ControlChannelPingerTask ] initialized
Oct 17, 2011 12:30:08 AM lia.util.net.copy.FDTReaderSession internalInit
INFO: [ FDTReaderSession ] No pre processing filters defined/processed.
Oct 17, 2011 12:30:09 AM lia.util.net.copy.FDTReaderSession startReading
INFO:  Started DiskReaderTasks for the following partions [ 64770 ] for FDTSession: 8d745066-e6f9-4fad-9446-eed7139f6221
Oct 17, 2011 12:30:09 AM lia.util.net.copy.transport.TCPTransportProvider tryToConnect
INFO: Requested window size -1. Using window size: 43690
17/10 00:30:18   Net Out: 7.374 Gb/s   Avg: 7.374 Gb/s
17/10 00:30:23   Net Out: 7.365 Gb/s   Avg: 7.369 Gb/s
17/10 00:30:28   Net Out: 7.324 Gb/s   Avg: 7.354 Gb/s
17/10 00:30:33   Net Out: 7.384 Gb/s   Avg: 7.361 Gb/s 67.75% ( 11s )
17/10 00:30:38   Net Out: 7.398 Gb/s   Avg: 7.368 Gb/s 82.11% ( 06s )
17/10 00:30:43   Net Out: 7.406 Gb/s   Avg: 7.374 Gb/s 96.47% ( 01s )
Oct 17, 2011 12:30:45 AM lia.util.net.copy.FDTReaderSession handleEndFDTSession
INFO: [ FDTReaderSession ] Remote FDTWriterSession for session [ 8d745066-e6f9-4fad-9446-eed7139f6221 ] finished OK!
17/10 00:30:48   Net Out: 1.816 Gb/s   Avg: 6.580 Gb/s 100.00% ( 00s )


FDTReaderSession ( 8d745066-e6f9-4fad-9446-eed7139f6221 ) final stats:
 Started: Mon Oct 17 00:30:08 PDT 2011
 Ended:   Mon Oct 17 00:30:48 PDT 2011
 Transfer period:   40s
 TotalBytes: 32212254720
 TotalNetworkBytes: 32212254720
 Exit Status: OK

Oct 17, 2011 12:30:48 AM lia.util.net.copy.FDTReaderSession doPostProcessing
INFO: [ FDTReaderSession ] Post Processing started
Oct 17, 2011 12:30:48 AM lia.util.net.copy.FDTReaderSession doPostProcessing
INFO: [ FDTReaderSession ] No post processing filters defined/processed.
 [ Mon Oct 17 00:30:49 PDT 2011 ] - GracefulStopper hook started ... Waiting for the cleanup to finish
 [ Mon Oct 17 00:30:49 PDT 2011 ]  - GracefulStopper hook finished!

 [ Mon Oct 17 00:30:49 PDT 2011 ]  FDT Session finished OK.

OCZ Deneva 2 Disk only Benchmark

Read from Disk to /dev/null 

[root@localhost ~]# java -cp /root/fdt/fdt.jar lia.util.net.common.DDCopy if=/deneva/10Gfile.dat of=/dev/null
[Mon Oct 17 00:43:39 PDT 2011] Current Speed = 720.584 MB/s Avg Speed: 720.944 MB/s Total Transfer: 1.409 GB
[Mon Oct 17 00:43:41 PDT 2011] Current Speed = 784.498 MB/s Avg Speed: 753.028 MB/s Total Transfer: 2.972 GB
[Mon Oct 17 00:43:43 PDT 2011] Current Speed = 794.955 MB/s Avg Speed: 766.918 MB/s Total Transfer: 4.526 GB
[Mon Oct 17 00:43:45 PDT 2011] Current Speed = 795.445 MB/s Avg Speed: 774.017 MB/s Total Transfer: 6.081 GB
[Mon Oct 17 00:43:47 PDT 2011] Current Speed = 794.84 MB/s Avg Speed: 778.166 MB/s Total Transfer: 7.635 GB
[Mon Oct 17 00:43:49 PDT 2011] Current Speed = 794.621 MB/s Avg Speed: 780.901 MB/s Total Transfer: 9.189 GB


 Total Transfer: 10 GBytes ( 10737418240 bytes )
 Time: 13 seconds
 Avg Speed: 781.441 MB/s

Note this is: 6 251 Mb/s

Write to disk from /dev/zero 

[root@localhost ~]# java -cp /root/fdt/fdt.jar lia.util.net.common.DDCopy if=/dev/zero of=/deneva/10Goutputfile5 bs=10M count=10240 
[Mon Oct 17 01:28:14 PDT 2011] Current Speed = 749.625 MB/s Avg Speed: 749.625 MB/s Total Transfer: 1.465 GB
[Mon Oct 17 01:28:16 PDT 2011] Current Speed = 955.882 MB/s Avg Speed: 853.749 MB/s Total Transfer: 3.369 GB
[Mon Oct 17 01:28:18 PDT 2011] Current Speed = 668.997 MB/s Avg Speed: 792.522 MB/s Total Transfer: 4.678 GB
[Mon Oct 17 01:28:20 PDT 2011] Current Speed = 654.346 MB/s Avg Speed: 758.141 MB/s Total Transfer: 5.957 GB
[Mon Oct 17 01:28:22 PDT 2011] Current Speed = 764.236 MB/s Avg Speed: 759.355 MB/s Total Transfer: 7.451 GB
[Mon Oct 17 01:28:24 PDT 2011] Current Speed = 928.607 MB/s Avg Speed: 787.487 MB/s Total Transfer: 9.268 GB
[snip]
[Mon Oct 17 01:30:38 PDT 2011] Current Speed = 729.271 MB/s Avg Speed: 661.099 MB/s Total Transfer: 94.395 GB
[Mon Oct 17 01:30:40 PDT 2011] Current Speed = 544.456 MB/s Avg Speed: 659.524 MB/s Total Transfer: 95.459 GB
[Mon Oct 17 01:30:42 PDT 2011] Current Speed = 689.311 MB/s Avg Speed: 659.921 MB/s Total Transfer: 96.807 GB
[Mon Oct 17 01:30:44 PDT 2011] Current Speed = 399.401 MB/s Avg Speed: 656.493 MB/s Total Transfer: 97.588 GB
[Mon Oct 17 01:30:46 PDT 2011] Current Speed = 644.034 MB/s Avg Speed: 656.331 MB/s Total Transfer: 98.848 GB


 Total Transfer: 100 GBytes ( 107374182400 bytes )
 Time: 155 seconds
 Avg Speed: 656.852 MB/s

OCZ Vertex 3 Network Transfers

Memory to Memory with FDT 

root@localhost fdt]# java -jar fdt.jar -c uvic-sonar.hepnetcanada.ca -bio -d /dev/null /dev/zero


FDT [ 0.9.23-201107290935 ] STARTED ... 


Oct 13, 2011 3:43:25 AM lia.util.net.common.Config <init>
INFO: Using lia.util.net.copy.PosixFSFileChannelProviderFactory as FileChannelProviderFactory
Oct 13, 2011 3:43:25 AM lia.util.net.common.Config <init>
INFO: FDT started in client mode
FDT uses *blocking* I/O mode.
Oct 13, 2011 3:43:25 AM lia.util.net.copy.transport.ControlChannel initStreams
INFO: NEW CONTROL stream for 270872f2-7d1f-4ae9-b176-366c4395ab18 initialized 
Oct 13, 2011 3:43:25 AM lia.util.net.copy.transport.ControlChannel$ControlChannelPingerTask <init>
INFO: [ ControlChannelPingerTask ] initialized
Oct 13, 2011 3:43:25 AM lia.util.net.copy.FDTReaderSession internalInit
INFO: [ FDTReaderSession ] No pre processing filters defined/processed.
Oct 13, 2011 3:43:26 AM lia.util.net.copy.FDTReaderSession startReading
INFO:  Started DiskReaderTasks for the following partions [ 0 ] for FDTSession: 270872f2-7d1f-4ae9-b176-366c4395ab18
Oct 13, 2011 3:43:26 AM lia.util.net.copy.transport.TCPTransportProvider tryToConnect
INFO: Requested window size -1. Using window size: 43690
13/10 03:43:35   Net Out: 9.897 Gb/s   Avg: 9.897 Gb/s
13/10 03:43:40   Net Out: 9.920 Gb/s   Avg: 9.908 Gb/s
13/10 03:43:45   Net Out: 9.879 Gb/s   Avg: 9.898 Gb/s
13/10 03:43:50   Net Out: 9.904 Gb/s   Avg: 9.899 Gb/s
[root@localhost fdt]#

SSD to Memory with FDT

  • Transfer from the OCZ disk and writing to /dev/null on the other machine
  • The drive was mounted and umounted prior to the test to eliminate anny cache issues. 
[root@localhost fdt]# java -jar fdt.jar -c uvic-sonar.hepnetcanada.ca -bio -d /dev/null /solidstate/10Gfile.dat 


FDT [ 0.9.23-201107290935 ] STARTED ... 


Oct 13, 2011 3:45:59 AM lia.util.net.common.Config <init>
INFO: Using lia.util.net.copy.PosixFSFileChannelProviderFactory as FileChannelProviderFactory
Oct 13, 2011 3:45:59 AM lia.util.net.common.Config <init>
INFO: FDT started in client mode
FDT uses *blocking* I/O mode.
Oct 13, 2011 3:45:59 AM lia.util.net.copy.transport.ControlChannel initStreams
INFO: NEW CONTROL stream for f72f0a43-79bb-4abc-ab69-4ad614fb4c5f initialized 
Oct 13, 2011 3:45:59 AM lia.util.net.copy.transport.ControlChannel$ControlChannelPingerTask <init>
INFO: [ ControlChannelPingerTask ] initialized
Oct 13, 2011 3:45:59 AM lia.util.net.copy.FDTReaderSession internalInit
INFO: [ FDTReaderSession ] No pre processing filters defined/processed.
Oct 13, 2011 3:46:00 AM lia.util.net.copy.FDTReaderSession startReading
INFO:  Started DiskReaderTasks for the following partions [ 64769 ] for FDTSession: f72f0a43-79bb-4abc-ab69-4ad614fb4c5f
Oct 13, 2011 3:46:00 AM lia.util.net.copy.transport.TCPTransportProvider tryToConnect
INFO: Requested window size -1. Using window size: 43690
13/10 03:46:09   Net Out: 2.962 Gb/s   Avg: 2.962 Gb/s
13/10 03:46:14   Net Out: 2.974 Gb/s   Avg: 2.968 Gb/s
13/10 03:46:19   Net Out: 2.974 Gb/s   Avg: 2.970 Gb/s
13/10 03:46:24   Net Out: 2.972 Gb/s   Avg: 2.970 Gb/s 82.16% ( 05s )
13/10 03:46:29   Net Out: 2.989 Gb/s   Avg: 2.974 Gb/s 99.57% ( 00s )
Oct 13, 2011 3:46:29 AM lia.util.net.copy.FDTReaderSession handleEndFDTSession
INFO: [ FDTReaderSession ] Remote FDTWriterSession for session [ f72f0a43-79bb-4abc-ab69-4ad614fb4c5f ] finished OK!


FDTReaderSession ( f72f0a43-79bb-4abc-ab69-4ad614fb4c5f ) final stats:
 Started: Thu Oct 13 03:45:59 PDT 2011
 Ended:   Thu Oct 13 03:46:32 PDT 2011
 Transfer period:   33s
 TotalBytes: 10737418240
 TotalNetworkBytes: 10737418240
 Exit Status: OK

Oct 13, 2011 3:46:32 AM lia.util.net.copy.FDTReaderSession doPostProcessing
INFO: [ FDTReaderSession ] Post Processing started
Oct 13, 2011 3:46:32 AM lia.util.net.copy.FDTReaderSession doPostProcessing
INFO: [ FDTReaderSession ] No post processing filters defined/processed.
 [ Thu Oct 13 03:46:33 PDT 2011 ] - GracefulStopper hook started ... Waiting for the cleanup to finish
 [ Thu Oct 13 03:46:33 PDT 2011 ]  - GracefulStopper hook finished!

 [ Thu Oct 13 03:46:33 PDT 2011 ]  FDT Session finished OK.

[root@localhost fdt]#

OCZ Vertex 3 Disk only Benchmark

  • Note that OCZ Vertex 3 is very similar performance to Deneva 2 (50 ordered)
  • Test below are using Dell R510 with H700 raid controller. Only a single disk is tested here.
  • SATA 3 disk connected to SATA 2 backplane.

Read from Disk to /dev/null 

[root@localhost fdt]# java -cp fdt.jar lia.util.net.common.DDCopy if=/solidstate/10Gfile.dat of=/dev/null
[Thu Oct 13 03:57:16 PDT 2011] Current Speed = 339.902 MB/s Avg Speed: 340.072 MB/s Total Transfer: 680.484 MB
[Thu Oct 13 03:57:18 PDT 2011] Current Speed = 354.596 MB/s Avg Speed: 347.404 MB/s Total Transfer: 1.371 GB
[Thu Oct 13 03:57:20 PDT 2011] Current Speed = 350.462 MB/s Avg Speed: 348.417 MB/s Total Transfer: 2.056 GB
[Thu Oct 13 03:57:22 PDT 2011] Current Speed = 356.111 MB/s Avg Speed: 350.332 MB/s Total Transfer: 2.752 GB
[Thu Oct 13 03:57:24 PDT 2011] Current Speed = 348.856 MB/s Avg Speed: 350.037 MB/s Total Transfer: 3.435 GB
[Thu Oct 13 03:57:26 PDT 2011] Current Speed = 353.125 MB/s Avg Speed: 350.551 MB/s Total Transfer: 4.125 GB
[Thu Oct 13 03:57:28 PDT 2011] Current Speed = 354.656 MB/s Avg Speed: 351.136 MB/s Total Transfer: 4.819 GB
[Thu Oct 13 03:57:30 PDT 2011] Current Speed = 355.207 MB/s Avg Speed: 351.643 MB/s Total Transfer: 5.514 GB
[Thu Oct 13 03:57:32 PDT 2011] Current Speed = 353.084 MB/s Avg Speed: 351.803 MB/s Total Transfer: 6.204 GB
[Thu Oct 13 03:57:34 PDT 2011] Current Speed = 353.834 MB/s Avg Speed: 352.006 MB/s Total Transfer: 6.896 GB
[Thu Oct 13 03:57:36 PDT 2011] Current Speed = 354.41 MB/s Avg Speed: 352.224 MB/s Total Transfer: 7.589 GB
[Thu Oct 13 03:57:38 PDT 2011] Current Speed = 351.395 MB/s Avg Speed: 352.155 MB/s Total Transfer: 8.276 GB
[Thu Oct 13 03:57:40 PDT 2011] Current Speed = 354.343 MB/s Avg Speed: 352.323 MB/s Total Transfer: 8.969 GB
[Thu Oct 13 03:57:42 PDT 2011] Current Speed = 355.092 MB/s Avg Speed: 352.521 MB/s Total Transfer: 9.664 GB


 Total Transfer: 10 GBytes ( 10737418240 bytes )
 Time: 29 seconds
 Avg Speed: 352.374 MB/s

Write to disk from /dev/zero 

[root@localhost ~]# java -cp /root/fdt/fdt.jar lia.util.net.common.DDCopy if=/dev/zero of=/solidstate/10Goutputfile5 bs=10M count=10240 
[Mon Oct 17 01:34:26 PDT 2011] Current Speed = 974.513 MB/s Avg Speed: 974.513 MB/s Total Transfer: 1.904 GB
[Mon Oct 17 01:34:28 PDT 2011] Current Speed = 1.114 GB/s Avg Speed: 1.038 GB/s Total Transfer: 4.443 GB
[Mon Oct 17 01:34:30 PDT 2011] Current Speed = 693.959 MB/s Avg Speed: 945.258 MB/s Total Transfer: 5.801 GB
[Mon Oct 17 01:34:32 PDT 2011] Current Speed = 389.416 MB/s Avg Speed: 810.909 MB/s Total Transfer: 6.562 GB
[Mon Oct 17 01:34:34 PDT 2011] Current Speed = 439.56 MB/s Avg Speed: 738.653 MB/s Total Transfer: 7.422 GB
[Mon Oct 17 01:34:36 PDT 2011] Current Speed = 454.319 MB/s Avg Speed: 692.32 MB/s Total Transfer: 8.311 GB
[Mon Oct 17 01:34:38 PDT 2011] Current Speed = 464.304 MB/s Avg Speed: 660.371 MB/s Total Transfer: 9.219 GB
[Mon Oct 17 01:34:40 PDT 2011] Current Speed = 389.416 MB/s Avg Speed: 627.071 MB/s Total Transfer: 9.98 GB
[Mon Oct 17 01:34:42 PDT 2011] Current Speed = 409.59 MB/s Avg Speed: 603.279 MB/s Total Transfer: 10.781 GB

[snip]

[Mon Oct 17 01:37:00 PDT 2011] Current Speed = 344.483 MB/s Avg Speed: 411.917 MB/s Total Transfer: 62.949 GB
[Mon Oct 17 01:37:02 PDT 2011] Current Speed = 369.446 MB/s Avg Speed: 411.38 MB/s Total Transfer: 63.672 GB
[Mon Oct 17 01:37:04 PDT 2011] Current Speed = 324.675 MB/s Avg Speed: 410.298 MB/s Total Transfer: 64.307 GB
[Mon Oct 17 01:37:06 PDT 2011] Current Speed = 404.393 MB/s Avg Speed: 410.225 MB/s Total Transfer: 65.098 GB
[Mon Oct 17 01:37:08 PDT 2011] Current Speed = 364.453 MB/s Avg Speed: 409.668 MB/s Total Transfer: 65.811 GB
[Mon Oct 17 01:37:10 PDT 2011] Current Speed = 349.476 MB/s Avg Speed: 408.944 MB/s Total Transfer: 66.494 GB
[Mon Oct 17 01:37:12 PDT 2011] Current Speed = 414.585 MB/s Avg Speed: 409.011 MB/s Total Transfer: 67.305 GB
[Mon Oct 17 01:37:14 PDT 2011] Current Speed = 359.461 MB/s Avg Speed: 408.429 MB/s Total Transfer: 68.008 GB
[Mon Oct 17 01:37:16 PDT 2011] Current Speed = 384.423 MB/s Avg Speed: 408.15 MB/s Total Transfer: 68.76 GB
[Mon Oct 17 01:37:18 PDT 2011] Current Speed = 339.491 MB/s Avg Speed: 407.362 MB/s Total Transfer: 69.424 GB
[Mon Oct 17 01:37:20 PDT 2011] Current Speed = 329.67 MB/s Avg Speed: 406.481 MB/s Total Transfer: 70.068 GB
[Mon Oct 17 01:37:22 PDT 2011] Current Speed = 399.401 MB/s Avg Speed: 406.402 MB/s Total Transfer: 70.85 GB
[Mon Oct 17 01:37:24 PDT 2011] Current Speed = 334.498 MB/s Avg Speed: 405.604 MB/s Total Transfer: 71.504 GB
[Mon Oct 17 01:37:26 PDT 2011] Current Speed = 429.356 MB/s Avg Speed: 405.864 MB/s Total Transfer: 72.344 GB
[Mon Oct 17 01:37:28 PDT 2011] Current Speed = 414.378 MB/s Avg Speed: 405.957 MB/s Total Transfer: 73.154 GB
[Mon Oct 17 01:37:30 PDT 2011] Current Speed = 279.72 MB/s Avg Speed: 404.602 MB/s Total Transfer: 73.701 GB
[Mon Oct 17 01:37:32 PDT 2011] Current Speed = 434.348 MB/s Avg Speed: 404.918 MB/s Total Transfer: 74.551 GB
[Mon Oct 17 01:37:34 PDT 2011] Current Speed = 344.483 MB/s Avg Speed: 404.283 MB/s Total Transfer: 75.225 GB
[Mon Oct 17 01:37:36 PDT 2011] Current Speed = 389.61 MB/s Avg Speed: 404.13 MB/s Total Transfer: 75.986 GB
[Mon Oct 17 01:37:38 PDT 2011] Current Speed = 354.468 MB/s Avg Speed: 403.619 MB/s Total Transfer: 76.68 GB
[Mon Oct 17 01:37:40 PDT 2011] Current Speed = 339.491 MB/s Avg Speed: 402.965 MB/s Total Transfer: 77.344 GB
[Mon Oct 17 01:37:42 PDT 2011] Current Speed = 424.363 MB/s Avg Speed: 403.181 MB/s Total Transfer: 78.174 GB
[Mon Oct 17 01:37:44 PDT 2011] Current Speed = 319.68 MB/s Avg Speed: 402.348 MB/s Total Transfer: 78.799 GB
[Mon Oct 17 01:37:46 PDT 2011] Current Speed = 399.401 MB/s Avg Speed: 402.318 MB/s Total Transfer: 79.58 GB
[Mon Oct 17 01:37:48 PDT 2011] Current Speed = 364.453 MB/s Avg Speed: 401.948 MB/s Total Transfer: 80.293 GB
[Mon Oct 17 01:37:50 PDT 2011] Current Speed = 349.476 MB/s Avg Speed: 401.439 MB/s Total Transfer: 80.977 GB
[Mon Oct 17 01:37:52 PDT 2011] Current Speed = 414.585 MB/s Avg Speed: 401.565 MB/s Total Transfer: 81.787 GB
[Mon Oct 17 01:37:54 PDT 2011] Current Speed = 344.483 MB/s Avg Speed: 401.022 MB/s Total Transfer: 82.461 GB
[Mon Oct 17 01:37:56 PDT 2011] Current Speed = 394.408 MB/s Avg Speed: 400.96 MB/s Total Transfer: 83.232 GB
[Mon Oct 17 01:37:58 PDT 2011] Current Speed = 349.476 MB/s Avg Speed: 400.479 MB/s Total Transfer: 83.916 GB
[Mon Oct 17 01:38:00 PDT 2011] Current Speed = 339.491 MB/s Avg Speed: 399.915 MB/s Total Transfer: 84.58 GB
[Mon Oct 17 01:38:02 PDT 2011] Current Speed = 374.625 MB/s Avg Speed: 399.683 MB/s Total Transfer: 85.312 GB
[Mon Oct 17 01:38:04 PDT 2011] Current Speed = 339.491 MB/s Avg Speed: 399.137 MB/s Total Transfer: 85.977 GB
[Mon Oct 17 01:38:06 PDT 2011] Current Speed = 389.416 MB/s Avg Speed: 399.049 MB/s Total Transfer: 86.738 GB
[Mon Oct 17 01:38:08 PDT 2011] Current Speed = 329.67 MB/s Avg Speed: 398.431 MB/s Total Transfer: 87.383 GB
[Mon Oct 17 01:38:10 PDT 2011] Current Speed = 334.498 MB/s Avg Speed: 397.866 MB/s Total Transfer: 88.037 GB
[Mon Oct 17 01:38:12 PDT 2011] Current Speed = 529.206 MB/s Avg Speed: 399.017 MB/s Total Transfer: 89.072 GB
[Mon Oct 17 01:38:14 PDT 2011] Current Speed = 329.506 MB/s Avg Speed: 398.413 MB/s Total Transfer: 89.717 GB
[Mon Oct 17 01:38:16 PDT 2011] Current Speed = 379.62 MB/s Avg Speed: 398.251 MB/s Total Transfer: 90.459 GB
[Mon Oct 17 01:38:18 PDT 2011] Current Speed = 339.491 MB/s Avg Speed: 397.749 MB/s Total Transfer: 91.123 GB
[Mon Oct 17 01:38:20 PDT 2011] Current Speed = 349.476 MB/s Avg Speed: 397.341 MB/s Total Transfer: 91.807 GB
[Mon Oct 17 01:38:22 PDT 2011] Current Speed = 419.58 MB/s Avg Speed: 397.527 MB/s Total Transfer: 92.627 GB
[Mon Oct 17 01:38:24 PDT 2011] Current Speed = 354.645 MB/s Avg Speed: 397.17 MB/s Total Transfer: 93.32 GB
[Mon Oct 17 01:38:26 PDT 2011] Current Speed = 399.6 MB/s Avg Speed: 397.19 MB/s Total Transfer: 94.102 GB
[Mon Oct 17 01:38:28 PDT 2011] Current Speed = 364.453 MB/s Avg Speed: 396.922 MB/s Total Transfer: 94.814 GB
[Mon Oct 17 01:38:30 PDT 2011] Current Speed = 334.498 MB/s Avg Speed: 396.415 MB/s Total Transfer: 95.469 GB
[Mon Oct 17 01:38:32 PDT 2011] Current Speed = 309.69 MB/s Avg Speed: 395.717 MB/s Total Transfer: 96.074 GB
[Mon Oct 17 01:38:34 PDT 2011] Current Speed = 359.461 MB/s Avg Speed: 395.427 MB/s Total Transfer: 96.777 GB
[Mon Oct 17 01:38:36 PDT 2011] Current Speed = 369.446 MB/s Avg Speed: 395.221 MB/s Total Transfer: 97.5 GB
[Mon Oct 17 01:38:38 PDT 2011] Current Speed = 359.461 MB/s Avg Speed: 394.94 MB/s Total Transfer: 98.203 GB
[Mon Oct 17 01:38:40 PDT 2011] Current Speed = 329.67 MB/s Avg Speed: 394.431 MB/s Total Transfer: 98.848 GB
[Mon Oct 17 01:38:42 PDT 2011] Current Speed = 414.585 MB/s Avg Speed: 394.587 MB/s Total Transfer: 99.658 GB


 Total Transfer: 100 GBytes ( 107374182400 bytes )
 Time: 259 seconds
 Avg Speed: 394.477 MB/s

IOZone (with unmount) 

   Iozone: Performance Test of File I/O
           Version $Revision: 1.7 $
      Compiled for 64 bit mode.
      Build: linux-AMD64 

   Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
                Ben England.

   Run began: Thu Oct 13 03:23:58 2011

   Excel chart generation enabled
   Auto Mode
   Using maximum file size of 2097152 kilobytes.
   Command line used: /root/iozone/iozone3_397/src/current/iozone -Ra -g 2G -f /solidstate/iozone/iozone -U /solidstate –i 0 –i 1
   Output is in Kbytes/sec
   Time Resolution = 0.000001 seconds.
   Processor cache size set to 1024 Kbytes.
   Processor cache line size set to 32 bytes.
   File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
              64       4  270108  551422   283217   259156   81324  552557   88161  1304314   743988   525513   542506  235303   247907
              64       8  344233  627470   312204   300665  154201  608965  146142  1599680   829001   592827   634890  225423   244075
              64      16  359444  639427   363829   365314  265565  603489  233461  1279447   674790  1226821  1232453  244297   246994
              64      32  367817  659861   789967   998622  318502  621657  292156  1518251   394870  1249666  1492919  273409   290890
              64      64  342040  666414  2923952  3057153  304761  546928  302018   610350   310759   639427   627470  296349   309326
             128       4  394814  723735   453896   470606   84217  702889   74546  1451764    99075   659704   744820  396271   383258
             128       8  444501  841746   430942   439767  136615  800337  159223  1155043  1229084   785122   761728  342216   403721
             128      16  435134  743788   540217   592699  258011  826202  216934  1852520  1093856   743788   841746  399812   429220
             128      32  368779  843068   579270   571257  328199  816154  339618  1905110   753179   877518   847059  460514   449337
             128      64  474349  770474  2969325  3359523  447464  789741  401307   881842   432679   847059   976473  445978   445978
             128     128  458548  919605  3199360  2511022  425141  673778  412721   836500   425141   921183   913347  428193   421139
             256       4  441359  921029   638066   633548   90391  785584   90239  1052888    92152   852984   886067  535583   565185
             256       8  473263  947860   531080   508929  160097  904732  161421  2045979   161493   948698   951219  585845   580461
             256      16  515032 1058076   764334   738065  267512  958864  270068  1938842  1619994  1003679   999940  449112   473054
             256      32  522551 1131677   723153   750970  380920 1048775  375459  2081678  1237306  1045710  1016025  625794   664935
             256      64  549283 1027695  1040643   907792  460476 1027695  457923  1463250   823544  1089202  1152323  484801   515032
             256     128  531080 1089202  2699161  2911402  649647  955451  537729  1334162   616807  1208072  1262029  635047   629094
             256     256  493264 1132871  3499745  3465855  591330  903970  577962  1108314   577651  1079348  1080434  583299   556687
             512       4  496206 1007827   837942   831132   97618  844865   96625  1394645    91985   903136   851903  691887   589028
             512       8  536092 1143004   644733   671129  167147 1032044  177715  2091421   160850   944444  1073309  814117   799865
             512      16  547575 1127402   894857   958781  284291 1080328  237479  2235086   272184  1089647   973118  654759   634448
             512      32  565751 1251589   788992   793951  377028 1169140  389403  2350044  1902438  1145443  1137555  820337   668413
             512      64  563968 1204552  1216149  1204552  515131 1190529  491887  1700565  1286067  1218910  1254513  665720   605811
             512     128  579179 1261144  1326585  1293037  644153 1127995  562638  1541840  1135150  1190529  1245780  751971   730484
             512     256  549959 1233614  1433753  1459080  694572 1064794  575917  1488408   928920  1361920  1450212  771143   701378
             512     512  517739 1282994  3710197  3580298  676201  939486  611678  1216149   665926  1209982  1258188  674714   677267
            1024       4  500765 1041758   857654   958739  101397  905384  100648   991266   101608   928479   964120  921705   746846
            1024       8  560495 1220575   742584   757382  168174 1035479  174984  2056183   158809  1071116  1113030  809484   782923
            1024      16  587562 1283303   821247   935966  290263 1182272  291563  1715375   256052  1150600  1195102  735086   746846
            1024      32  601973 1296083   853394   885778  410930 1291018  421537  2198257   358278  1207534  1185209  821090   776413
            1024      64  598117 1349004  1009676  1203137  529210 1281771  485098  2156318  2077065  1142641  1258117  785788   782210
            1024     128  593899 1315938  1001903  1082454  620229 1214706  584762  2004366  1809317  1248970  1322828  810095   721624
            1024     256  514810 1369216  1111877  1140214  715135 1229309  606650  1835608  1602118  1347311  1387798  811472   799390
            1024     512  613934 1385560  1464956  1471481  794363 1126458  667977  1592612  1102176  1508167  1582636  770562   778665
            1024    1024  613846 1345623  3643490  3643490  711699 1027060  693317  1340582   719087  1363133  1365733  690975   722230
            2048       4  529595 1055170   829480  1060511   99036  930471  101921  1515833    98846   983639   981728  773672   772211
            2048       8  571754 1196902   846648   719075  178615 1116038  174281  2052157   182823  1124071  1123336  864108   854736
            2048      16  592253 1214673   936865   919121  296554 1240458  295351  2290767   264461  1200414  1212444  826129   833585
            2048      32  586992 1368084  1037957   906990  427024 1296045  422279  2467800   380105  1218982  1257344  950339   900147
            2048      64  617412 1441069  1023487   958825  550961 1339497  552449  2345819   583325  1266427  1303716  835937   911417
            2048     128  596241 1447382   996534  1008941  631714 1307885  623235  2270183  2522144  1314489  1259557  871472   868564
            2048     256  617589 1462911  1048600  1100595  668851 1334295  626508  2111168  2142232  1294677  1377297  846231   852277
            2048     512  601333 1469167  1115459  1197736  811764 1300361  721127  2008023  1765367  1391351  1473451  832454   838631
            2048    1024  621746 1432179  1095682  1159578  793175 1193576  740715  1690996  1181915  1533970  1648801  786493   793468
            2048    2048  631343 1418228  3757946  3677503  723435 1000015  739886  1371798   729580  1420104  1445434  737789   731693
            4096       4  529347 1096927   824971   803931  102094  974312  104476  1656241   102195  1035692  1016391  796108   843150
            4096       8  579347 1206806   846307   902977  181077 1101075  182840  2167039   182547  1166333  1148632  881796   933241
            4096      16  600761 1345140   914075   981549  294207 1286900  293262  2367741   305741  1211060  1237047  882566   943388
            4096      32  606552 1414352   943958   942922  430758 1344192  433202  2478054   373452  1239725  1289314 1002220  1007392
            4096      64  613636 1407055  1030907  1142825  563499 1307368  561327  2415003   570801  1331999  1324196  991519   938800
            4096     128  635825 1475071  1020921  1143814  639089 1412492  626778  2387152   690375  1293294  1311960  913055   976861
            4096     256  580581 1475958  1102205  1160660  733906 1406594  633270  2287049  2900630  1298180  1336351  978474  1000003
            4096     512  603145 1492758  1134973  1190664  839852 1409479  767616  2253155  2459958  1403262  1342932  924404   918424
            4096    1024  615659 1471785   852480   898021  956573 1345140  858358  2081448  1811495  1322565  1326036  935783   962414
            4096    2048  607796 1470274  1158078  1235268  877652 1204775  826638  1755052  1207146  1497051  1581968  904593   855451
            4096    4096  613001 1481942  3565171  3546770  836621 1038008  741892  1425146   772691  1412376  1447968  753080   795776
            8192       4  502513 1053210   950245   990827  102047  920955  105306  1650259   101713   985570   973727  941393  1014495
            8192       8  574387 1204556   990913  1071003  179095 1108707  178090  2155187   179977  1149277  1138576 1012492  1070570
            8192      16  579844 1296407  1060328  1125117  297188 1247231  297263  2343166   301274  1208581  1204176 1007622  1067510
            8192      32  609703 1374510  1134778  1140125  433573 1305123  436953  2546522   443118  1223776  1233928 1058401  1106422
            8192      64  624338 1370343  1089512  1139218  574473 1347820  573869  2564005   573706  1266540  1281180 1024629  1130559
            8192     128  577389 1379587  1082885  1137483  668401 1393462  665051  2485368   683003  1264955  1251273 1026435  1086171
            8192     256  629082 1390361  1195753  1184173  769211 1359016  660652  2384800   987325  1280035  1280798 1088649  1110463
            8192     512  603238 1418721  1176832  1227098  918910 1388227  831173  2337110  3047749  1288870  1311350 1086308  1129184
            8192    1024  612300 1403709   826972   875947  971634 1307955  933541  2195116  2511345  1199426  1212890 1050666  1093604
            8192    2048  620225 1401991   960364  1021310 1000493 1273441  951692  2095130  1860118  1309251  1303638  989600  1033568
            8192    4096  608483 1372917  1367181  1409468  964678 1128999  909521  1723173  1211308  1508673  1593472  905280   964922
            8192    8192  601559 1317888  3086904  3098317  919821  949693  865423  1307358   868991  1321487  1294600  843229   914581
           16384       4  515382 1038877  1030635  1098934   95631  900025  102555  1635290    99598   985793   992298 1045023  1013728
           16384       8  560249 1202760  1106187  1134624  169471 1080107  182011  2206308   177244  1110244  1113121 1112365  1121185
           16384      16  583269 1253574  1143974  1180142  282755 1218499  295265  2314113   287041  1166856  1142928 1112455  1137931
           16384      32  589923 1337584  1148621  1183312  419037 1282515  440690  2508210   439178  1230389  1206836 1152899  1164050
           16384      64  589478 1385897  1171451  1170872  554109 1319706  573327  2514451   576898  1216148  1235966 1147738  1174816
           16384     128  597587 1347815  1149504  1193838  671089 1336907  675151  2524891   678370  1213100  1268500 1139120  1140310
           16384     256  611868 1383999  1202507  1194876  800470 1372308  678873  2395675   991998  1243953  1267727 1160668  1189148
           16384     512  613239 1405886  1208343  1223140  928907 1333457  854806  2382056  1102266  1280197  1266046 1187689  1174735
           16384    1024  621385 1398562   845187   879071 1042834 1346495  993159  2358753  3097731  1160668  1140784 1124340  1178867
           16384    2048  619949 1351022   914287   934046 1066314 1300721 1049860  2290816  2523685  1190178  1185701 1103611  1132120
           16384    4096  596648 1314003  1048979  1061406 1081382 1250586 1052675  1995582  1850491  1286139  1284721 1067026  1094384
           16384    8192  596089 1258672  1323952  1324487 1045563 1093879 1004482  1555943  1305515  1402615  1519139 1006630  1036308
           16384   16384  568574 1268312  2965394  2903377  986699  942049  955169  1256324   963877  1284000  1232530  957005   977742
           32768      64  607986 1361876  1206739  1207322  543029 1336151  577004  2640426   571769  1251358  1236764 1223552  1231832
           32768     128  627366 1395562  1224970  1225559  672688 1331710  687463  2568433   687638  1250493  1248970 1202937  1219048
           32768     256  606714 1400041  1235686  1229056  805409 1325405  687418  2479603   983788  1232992  1229715 1216695  1224151
           32768     512  610487 1368466  1249879  1263181  972834 1348871  876452  2395500  1102883  1268251  1267772 1204740  1211163
           32768    1024  596585 1390775   849286   863338 1061552 1360743 1029013  2438122  1203822  1126550  1144620 1211344  1207810
           32768    2048  590277 1395520   884951   897678 1123768 1346097 1092486  2381057  3098625  1144858  1140195 1192003  1201307
           32768    4096  590021 1357250   952837   933214 1129736 1257161 1114754  2221422  2454494  1170541  1174523 1165825  1178410
           32768    8192  594925 1266055  1044267  1046406 1120498 1168859 1100296  1722361  1850137  1234432  1254717 1118137  1138693
           32768   16384  595110 1275762  1316404  1313661 1097502 1072503 1070640  1513531  1413809  1417966  1492639 1067389  1092078
           65536      64  593606 1350978  1215790  1230309  550219 1336735  568720  2680407   558665  1234292  1246499 1244862  1244062
           65536     128  583205 1350288  1231698  1227194  666679 1301401  683251  2586774   671200  1240531  1236713 1238373  1248991
           65536     256  596453 1388210  1240671  1254733  788619 1390035  688736  2472595   986053  1252434  1238915 1242836  1248039
           65536     512  617589 1389038  1254710  1263685  952199 1386823  889863  2478391  1088548  1263586  1238117 1239429  1234918
           65536    1024  609727 1368124   855683   864431 1085895 1380430 1044997  2496624  1174878  1143415  1144891 1249997  1247495
           65536    2048  616346 1399740   874244   877545 1140312 1368009 1135531  2454052  1242342  1140350  1134233 1232300  1238228
           65536    4096  582719 1354720   900790   902523 1177223 1314879 1158371  2264143  3056990  1130492  1123097 1211722  1223001
           65536    8192  578214 1259072   928968   941366 1159661 1188857 1147180  1890120  2305891  1154901  1147501 1180768  1180509
           65536   16384  587930 1256562  1036208  1036024 1147218 1169301 1138386  1690986  1915982  1242999  1249951 1156062  1168471
          131072      64  611295 1385742  1229290  1229524  543132 1357653  569563  2672547   556420  1239040  1232355 1257792  1266115
          131072     128  591528 1377720  1265479  1263879  663974 1352486  687262  2535004   673632  1249493  1232181 1257082  1258023
          131072     256  590794 1381366  1264623  1260645  772533 1366872  685619  2416957   995496  1238058  1229893 1255612  1258962
          131072     512  615934 1411973  1276732  1274238  928955 1381786  894885  2488125  1112769  1268343  1250366 1275814  1274657
          131072    1024  666151 1417883   856746   859493 1082505 1400088 1043269  2506925  1189098  1146906  1134174 1260151  1261932
          131072    2048  598646 1399561   863976   871356 1158935 1419648 1142746  2490063  1231949  1148423  1140042 1259267  1250508
          131072    4096  592855 1379283   882240   882171 1187934 1353895 1180968  2372420  1261594  1111711  1125642 1246417  1249493
          131072    8192  582094 1283622   898374   894463 1183398 1221628 1174669  1971844  2677441  1109331  1089922 1203942  1207957
          131072   16384  571582 1251641   936162   936315 1186227 1222071 1180922  1836407  2334441  1172672  1155449 1198305  1205713
          262144      64  607643 2426675  1263782  1239288  539375 1320792  566160  2715978   553815  1250304  1234036 1266078  1266365
          262144     128  599242 2391954  1308195  1257218  653246 1362007  684175  2583233   670095  1261314  1210900 1265851  1270367
          262144     256  599572 2427259  1257730  1253186  776177 1366136  693235  2467796  1001256  1273613  1274350 1269364  1258268
          262144     512  594394 2453999  1369971  1347388  960723 1406487  894710  2360020  1119785  1271703  1300961 1273848  1267638
          262144    1024  602097 2487088   856769   856741 1056117 1431488 1048815  2504571  1201223  1130769  1212036 1314953  1312762
          262144    2048  594216 1504109   842329   854969 1154275 1390300 1159482  2460380  1233787  1126062  1140113 1271667  1273415
          262144    4096  611226 1468692   864052   866073 1202893 1373999 1190897  2338089  1239070  1127109  1140421 1263093  1259332
          262144    8192  558858 2055758   881488   873667 1184804 1257406 1190919  1923484  1232806  1112410  1148214 1256899  1243344
          262144   16384  594662 1309653   874722   881859 1198361 1213988 1199744  1861421  2619813  1097585  1683465 1284251  1271222
          524288      64  875517 1366528  1180743  1250774  545850 1300074  570160  2587607   557219  1256612  1309882 1288341  1308413
          524288     128  620499 1285016  1279210  1276208  664846 1364902  685698  2529200   672451  1250164  1391846 1325240  1306135
          524288     256  725175 1307044  1397143  1372165  781097 1382393  688305  2472730  1002518  1243228  1448609 1322043  1292836
          524288     512  713398 1374647  1286502  1290011  935158 1426835  877616  2521535  1137947  1280706  1394938 1313985  1274542
          524288    1024  608024 1355320   854653   865215 1056452 1415391 1043791  2484730  1215582  1159398  1231732 1190251  1290148
          524288    2048  640725 1296420   859666   863256 1164251 1386565 1180154  2454645  1240155  1149976  1294704 1230033  1291973
          524288    4096  642401 1387688   858098   868893 1219438 1394215 1208398  2426910  1243375  1127654  1175196 1115840  1271698
          524288    8192  702171 1270696   857837   870833 1202263 1236032 1087847  2025286  1225238  1111752  1213045 1262128  1252486
          524288   16384  699100 1664073   884980   886834 1220117 1245981 1117270  1926431  1268844  1110445  1221729 1057375  1238617
         1048576      64  891092 1473574   610956   922549  449705 1347422  461988  2577664   379813  1375340  1090918  810929   799376
         1048576     128  841309 1405807   817896  1004812  560858 1371982  542012  2631816   567770  1323648  1066833  814900  1000289
         1048576     256  892541 1444168   828310  1005346  658566 1603366  549977  2444022   799747  1281600  1054080  786911  1000302
         1048576     512  796230 1426647   831996  1005376  750815 1419430  689174  2412627   906757  1301411  1089884  826226  1011069
         1048576    1024  857969 1465451   616235   714719  823538 1439906  759350  2470364   947887  1196742  1011545  820724  1006585
         1048576    2048  928603 1509031   602405   710421  899080 1447599  791273  2411290   962278  1180790   991273  829295  1009821
         1048576    4096  794835 1409838   651277   717130  925019 1435236  837362  2431109   965117  1151271   977403  818273  1010663
         1048576    8192  757091 1315069   673872   682695  911588 1508599  808442  2031352   953788  1202714   975848  804692   988478
         1048576   16384  792421 1390966   687762   731398  904403 1321475  827509  1896626   930582  1155238  1002387  688125   980680
         2097152      64 1031171 1770627   390372   564365  292022 1603349  274707  2831655   289663  1351395  1350730  498634   572438
         2097152     128 1017715 1704943   426329   580950  359448 1604318  322643  2609001   362871  1335615  1958737  519137   496266
         2097152     256  973676 1937322   466820   557459  405057 1560478  320586  2711193   451606  1334837  2007062  462471   579897
         2097152     512 1072119 1766945   400118   557173  434214 1633085  366097  2810506   509328  1376968  2063719  446724   510034
         2097152    1024  971365 1964839   344051   431802  459502 1561453  400366  2768945   536033  1248424  1202292  448794   503114
         2097152    2048 1025987 1711640   345014   436114  511696 1508147  418939  2593225   539765  1296243  1170319  415971   532342
         2097152    4096 1008702 1663828   342256   432258  529240 1506759  436950  2608375   539576  1209532  1132372  521643   454768
         2097152    8192  926850 1590699   343224   435083  535003 1339711  535224  2130810   541344  1183867  1116827  518469   485655
         2097152   16384  972598 1637733   345430   438004  553267 1348179  537660  2162128   540044  1201377  1091806  451251   573869

iozone test complete.

Network Host

TCP Tunning 

# increase Linux TCP buffer limits
net.core.rmem_max = 335544320
net.core.wmem_max = 335544320

# Default [rw]_mem_default 110592
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216

# increase Linux autotuning TCP buffer limits
net.ipv4.tcp_rmem = 4096 87380 335544320
net.ipv4.tcp_wmem = 4096 87380 335544320

# don't cache ssthresh from previous connection
net.ipv4.tcp_no_metrics_save = 1
# recommended to increase this for 1000 BT or higher
net.core.netdev_max_backlog = 2500

# Disable SACK ??
#net.ipv4.tcp_sack = 0
#net.ipv4.tcp_timestamps = 0

# Enable Explicit Congestion Notification
net.ipv4.tcp_ecn=1

# Explicitly set CUBIC as the congestion control
net.ipv4.tcp_congestion_control=cubic

-- IanGable - 2011-10-13

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf SC11_NOC_Westin_VICTX_UVic.pdf r1 manage 328.0 K 2011-10-13 - 16:46 UnknownUser UVic 100G network Diagram
This topic: HEPrc > WebHome > HundredGigabit
Topic revision: r7 - 2011-10-21 - igable
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback