Difference: HundredGigabitTestingLog (7 vs. 8)

Revision 82011-11-09 - igable

Line: 1 to 1
 
META TOPICPARENT name="HundredGigabit"

November 8, 2011

Line: 7 to 7
  100G-2011-11-08.png
Added:
>
>
'

Files produced via dd and /dev/zero has some interesting properties when stored on the SSDs. It would appear that writing zero to a ssd gives an artificially good results. We haven't discovered the cause of this, but it definitely has in impact. We switched to using files produced by /dev/urandom our disk to disk performance has deteriorated significantly:

5 drive config D to D 6 Drive config D to D 5 Drive D to M 6 Drive D to M
Avg: 5.536 Gb/s Avg: 6.172 Gb/s Avg: 8.525 Gb/s Avg: 9.787 Gb/s

In an effort to get this back to something more reasonable we pulled two drives out of scdemo08 and placed them in scdemo06 and scdemo07. We improved performance by about 12 percent. (We might be able to add a few more drives).

We did some more investigation and discovered that we get different results doing pure read of all zero files vs random files. For example:

Read a random file: 955.402 MB/s 7.46 Gb/s
Read a zeroed file: 1.205 GB/s 9.64 Gb / s
Write a /dev/zero file: 1.372 GB/s 10.97600 Gb/s

scdemo07->scdemo00
Avg: 5.536 Gb/s 

FDTReaderSession ( 2df13142-1621-496a-96be-4ded64eb9645 ) final stats:
 Started: Tue Nov 08 17:10:41 PST 2011
 Ended:   Tue Nov 08 17:23:37 PST 2011
 Transfer period:   12m 56s
 TotalBytes: 536870913830
 TotalNetworkBytes: 536870913830
 Exit Status: OK


scdemo06->scdemo09
Avg: 6.172 Gb/s 100.00% ( 00s )


FDTReaderSession ( 7abaf554-401a-4ba3-85d0-16fe0eae35ab ) final stats:
 Started: Tue Nov 08 17:02:29 PST 2011
 Ended:   Tue Nov 08 17:14:05 PST 2011
 Transfer period:   11m 35s
 TotalBytes: 536870913830
 TotalNetworkBytes: 536870913830
 Exit Status: OK

Writing to disk with with the /dev/zero device:

[iscdemo00 ~]$ java -cp ~/fdt/fdt.jar lia.util.net.common.DDCopy if=/dev/zero of=/ssd/10Goutputfile5 bs=10M count=10240
[Tue Nov 08 19:09:03 PST 2011] Current Speed = 1.416 GB/s Avg Speed: 1.416 GB/s Total Transfer: 2.832 GB
[Tue Nov 08 19:09:05 PST 2011] Current Speed = 1.328 GB/s Avg Speed: 1.371 GB/s Total Transfer: 5.605 GB
[Tue Nov 08 19:09:07 PST 2011] Current Speed = 1.313 GB/s Avg Speed: 1.352 GB/s Total Transfer: 8.232 GB
[Tue Nov 08 19:09:09 PST 2011] Current Speed = 1.309 GB/s Avg Speed: 1.341 GB/s Total Transfer: 10.85 GB
[Tue Nov 08 19:09:11 PST 2011] Current Speed = 1.284 GB/s Avg Speed: 1.33 GB/s Total Transfer: 13.418 GB
[Tue Nov 08 19:09:13 PST 2011] Current Speed = 1.401 GB/s Avg Speed: 1.342 GB/s Total Transfer: 16.221 GB
[Tue Nov 08 19:09:15 PST 2011] Current Speed = 1.401 GB/s Avg Speed: 1.35 GB/s Total Transfer: 19.023 GB
[Tue Nov 08 19:09:17 PST 2011] Current Speed = 1.391 GB/s Avg Speed: 1.355 GB/s Total Transfer: 21.816 GB
[Tue Nov 08 19:09:19 PST 2011] Current Speed = 1.392 GB/s Avg Speed: 1.359 GB/s Total Transfer: 24.6 GB
[Tue Nov 08 19:09:21 PST 2011] Current Speed = 1.396 GB/s Avg Speed: 1.363 GB/s Total Transfer: 27.393 GB
[Tue Nov 08 19:09:23 PST 2011] Current Speed = 1.396 GB/s Avg Speed: 1.366 GB/s Total Transfer: 30.186 GB
[Tue Nov 08 19:09:25 PST 2011] Current Speed = 1.387 GB/s Avg Speed: 1.368 GB/s Total Transfer: 32.959 GB
[Tue Nov 08 19:09:27 PST 2011] Current Speed = 1.391 GB/s Avg Speed: 1.369 GB/s Total Transfer: 35.742 GB
[Tue Nov 08 19:09:29 PST 2011] Current Speed = 1.382 GB/s Avg Speed: 1.37 GB/s Total Transfer: 38.506 GB
[Tue Nov 08 19:09:31 PST 2011] Current Speed = 1.386 GB/s Avg Speed: 1.371 GB/s Total Transfer: 41.279 GB
[Tue Nov 08 19:09:33 PST 2011] Current Speed = 1.377 GB/s Avg Speed: 1.372 GB/s Total Transfer: 44.033 GB
[Tue Nov 08 19:09:35 PST 2011] Current Speed = 1.381 GB/s Avg Speed: 1.372 GB/s Total Transfer: 46.797 GB
[Tue Nov 08 19:09:37 PST 2011] Current Speed = 1.382 GB/s Avg Speed: 1.373 GB/s Total Transfer: 49.561 GB
[Tue Nov 08 19:09:39 PST 2011] Current Speed = 1.357 GB/s Avg Speed: 1.372 GB/s Total Transfer: 52.275 GB
^C

 Total Transfer: 52.812 GBytes ( 56706990080 bytes )
 Time: 38 seconds
 Avg Speed: 1.372 GB/s

read a random:

[igable@scdemo00 rbatch0]$ java -cp ~/fdt/fdt.jar lia.util.net.common.DDCopy if=/ssd/10Grandom03.dat of=/dev/null bs=10M count=10240
Got exception: 
java.io.FileNotFoundException: /ssd/10Grandom03.dat (No such file or directory)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(Unknown Source)
   at java.io.RandomAccessFile.<init>(Unknown Source)
   at lia.util.net.common.DDCopy.main(DDCopy.java:371)
[igable@scdemo00 rbatch0]$ java -cp ~/fdt/fdt.jar lia.util.net.common.DDCopy if=/ssd/rbatch0/10Grandom03.dat of=/dev/null bs=10M count=10240
[Tue Nov 08 19:16:10 PST 2011] Current Speed = 760 MB/s Avg Speed: 760 MB/s Total Transfer: 1.484 GB
[Tue Nov 08 19:16:12 PST 2011] Current Speed = 1,008.646 MB/s Avg Speed: 886.82 MB/s Total Transfer: 3.535 GB
[Tue Nov 08 19:16:14 PST 2011] Current Speed = 989.505 MB/s Avg Speed: 920.598 MB/s Total Transfer: 5.469 GB
[Tue Nov 08 19:16:16 PST 2011] Current Speed = 990 MB/s Avg Speed: 937.771 MB/s Total Transfer: 7.402 GB
[Tue Nov 08 19:16:18 PST 2011] Current Speed = 1,000 MB/s Avg Speed: 950.114 MB/s Total Transfer: 9.355 GB


 Total Transfer: 10 GBytes ( 10737418240 bytes )
 Time: 10 seconds
 Avg Speed: 955.402 MB/s

Now read a file made with /dev/zero:

[igable@scdemo00 rbatch0]$ java -cp ~/fdt/fdt.jar lia.util.net.common.DDCopy if=/ssd/rbatch0/10Gfile01.dat of=/dev/null 
[Tue Nov 08 19:40:14 PST 2011] Current Speed = 1.163 GB/s Avg Speed: 1.164 GB/s Total Transfer: 2.328 GB
[Tue Nov 08 19:40:16 PST 2011] Current Speed = 1.194 GB/s Avg Speed: 1.179 GB/s Total Transfer: 4.753 GB
[Tue Nov 08 19:40:18 PST 2011] Current Speed = 1.229 GB/s Avg Speed: 1.196 GB/s Total Transfer: 7.211 GB
[Tue Nov 08 19:40:20 PST 2011] Current Speed = 1.231 GB/s Avg Speed: 1.204 GB/s Total Transfer: 9.674 GB


 Total Transfer: 10 GBytes ( 10737418240 bytes )
 Time: 8 seconds
 Avg Speed: 1.205 GB/s

 

November 7, 2011

All nodes are now configured with the Write Through on the SSD logical disk. SNMP polling to the Brocade now done in 1 min intervals.

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback