File SubSystem Benchmarks with Xen and iSCSI

I took some time to analyze the differences in file system i/o speed in kilobytes on local and iscsi san storage over copper and fiber, mounting them in the Xen Dom0 and also in the Xen domU.  All domU's are CentOS 5.x x64.  Tests were run on Windows Server R2 2003 Standard, but the results are extremely flawed.  I am suspecting this is due to some inadequate timing within Xen HVM.  As a sidenote, the fastest test completion was when the Dom0 had the iSCSI target mapped directly to a block device (/dev/sdi), and the domU config used the phy:/dev/sdi directive.  Recently, there has been a script released that will allow phy to be replaced with iscsi.  I tried to use it, unfortunatly it cause the domU's to be unable to boot.

The servers:

Xen Dom0:

• Intel S5000VSA-SATA Motherboard 
• (1) Xeon 5160 CPU
• 4 GB Kingston 667 FBDimm (2x2gb)
• 2u Supermicro w/ sas backplane
• 4 sata2 hard drives made by western digital (Raid Edition) w/ 16mb cache
• Intel SC Fiber Card
• Onboard Copper GB Nics
• OS: CentOs 5.0 updated to 5.1 x64 edition w/ Virtualization

Xen DomU:

• 768 GB RAM
• 4 GB lvm slice on local host for boot
• 4 GB iSCSI target from San01 (where noted iSCSI w/ applicable drive information)

San01:

• Tyan 2865 Motherboard
• Opteron 180 CPU
• 2 GB Kingston ECC 400mhz DDR
• RaidCore 4852 8 port sata drives
• 8 Seagate 300GB Sata drives in Raid 50 (submembers of 4)
• 4 Western Digital Raptors software Raid 5
• Intel LC Fiber Card
• Onboard Copper Nics (Broadcom was used in tests)
• 2 GB CF card on ide to cf converter (Boot Drive)
• OS: Openfiler x64

The Network:

Fiber connections are made through a 4 port fiber module (with 2 ports in use San01 and Dom0) on a Foundry FastIron II Switching Router.

Copper connections are made through a Dell 2716 Switch.

For the iSCSI initiator, I used iscsi-initiator-utils package installed by yum on CentOS 5.1 x64 edition.  This is the case with both the Xen Dom0 and Xen DomU's.  All of the settings are set to defaults.  By using some of these graphs to see how various storage subsystems perform with various block sizes for the array and iSCSI transfer, you can tweak the settings to met the needs of your application or cluster environment.

For the target, Openfiler uses Enterprise iSCSI Target (IET).  You can grab a copy for yourself on sourceforge.

All of these tests were compiled using IoZone.  This is a great utility made by a nice fella.  Check out the site for more details.  If you would like to see more graphs from these runs, contact me using the contact us link on the left of the page.

First, check out these results from the local storage subsystem on the Xen Dom0.  This test was run in the Xen Dom0 on the local array.  Remember these results, as you may notice similarities in later graphs, even if its pointed at an iSCSI target.

Click the image(s) to open it in a new window.

The next results are quite fascinating.  To me, it appears the Dom0 is caching much of the writes that are meant to go on the iSCSI target when the target is actually mounted in the Dom0, and passed to the DomU in the conifguration file, as /dev/sda.

Click the image(s) to open it in a new window.

This next graph is a kicker!  There is really no way possible that the hardware behind this could support that kind of sustained transfer.  This must be a result of the caching performed by xen itself.

Again, this next graph must be the result of some intellegent caching.  The underlying hardware is not supposed to support transfers of this speed. 

Before I forget, I want to add some verbage regarding the other resources that were consumed during this test.  The san01 openfiler server was carrying a load of 3.00 3.00 2.96 or so, and the cpu was seeing spikes up to around 12%.  CPU stayed quite steady around 3-5% otherwise.  The ram got full fast, but flushed frequently to the disk.  I advise using as much ram as you can, as this will cache the received iSCSI commands until they can flush to disk.