After spending what now amounts to 4 weeks dealing with this issue. Get ready for a long post hopefully with lots of performance/throughput numbers.
The challenge:
Windows 2000 SP4 Guest
>512MB RAM
>2 CPU (tried with 1 in many test also, just prolonged the inevitable)
4GB OS vmdk (IDE)
>Separate 12GB (tried 20GB also) persistent, pre-allocated SCSI drive, 2GB Files (tried as one file, no diff) >Running IOMeter 2004.07.30 on the full 12GB drive, 100% Sequential 64k Writes (or Reads, depending on test) Continuous test, watching "Results Since" at 1 sec update and getting a "mental" average watching for extremes that might be averaged out.
The hardware:
Supermicro X7DBE motherboard
2x Xeon Quad Core 2.33GHz E5345 (8 cores total)
8GB RAM
2 Raptor 74GB drives (mirrored for system)
6 73GB 15k SAS drives connected to Adapted 3805 SAS controller
Running in RAID5 (64k stripe, I think, maybe 128k, yes I know RAID10 is faster, but I'd prefer the space and as you'll see below, it should be plenty fast)
The goal: I want to run this server under a variant of linux, preferably Fedora Core 6 (FC6), CentOS 5, or Ubuntu 6.06 LTS (Preferably a 2.6.18 kernel, since that's required for LM sensors to work with that board (Actually .20 is required, but the module backports and compiles to .18 just fine, FYI) I want it on linux for several reasons: licensing, flexibility, DRBD, higher performance*, etc.
The problem: Windows seems to have way better disk performance in guests than linux, not good...
VMWare on Windows Server 2003 R2 x64
Host Disk Performance (IOMeter 2004.07.30), same settings as guest above:
220MB/s read, 150MB/s write (fyi 1MB request size bumps up to 315MB/s & 195MB/s) This is sustained, the initial burst (and this goes for all the host tests, and some of the guest test as well) can peak up to 2GB/s, some run at 500-600MB/s for a several seconds before dropping down. FYI 4k 100% rnd read 250iops read 700iops write (??!?)
Windows version of dd writes 12GB file to disk at 258MB/s
2000 Guest under Win2k3
Read: 180-247 Avg 230MB/s
Write: 110-122 Avg 122MB/s
Very good, that's what I was expecting, on to the linux tests:
Tried FC5 x64 yum updated with latest kernel 2.6.20, BAD MOVE, as well documented in the forum, 2.6.19 and greater the vm (and machine) will crash/lock up. Need to stay with 2.6.18 or lower (though 2.6.16 and 2.6.17 have other problems I've read of on the forums)
CentOS 4.4 x64 (yum updated)
Host Disk Performance (iometer slaves running under linux, manager running on windows)
226MB/s read, 170MB/s write (better than windows here, all linuxes performed similarly in host testing)
2000 Guest under CentOS 4.4 x64 2.6.9-55? kernel
Read: Avg 240MB/s (But unstable)
Write: Avg 38MB/s ?!!!!!!!?!???! Thus starts the problems (very unstable)
Sisoft Sandra 1117XP 2CPU 14298MIPS/13986MFLOPS (running in guest)
Decided to run with this anyway, but when I converted as server over it had weird "pausing" problems, evidenced by the clock VISIBLY stopping and loosing time (a several seconds per MINUTE!) Moved host over to windows server so I could reformat and continue testing.
FC6 x64 (NO kernel update, stick with stock 2.6.18)
Host Disk Performance (same as CentOS), still good
2000 Guest under FC6 2.6.18 kernel
Read: Avg 50MB/s (But unstable)
Write: Avg 20MB/s (very unstable goes to 0 for several seconds some times, CentOS did same thing, vmware logs "Command WRITE(10) took N.NNN seconds" up to 15 seconds)
Sisoft Sandra 1117XP 2CPU 20804MIPS/14271MFLOPS (running in guest), newer kernels have better cpu performance in guests.
Better performance in Sandra, feels a little snappier too, but disk performance is still poor, see conclusions below.
UBuntu 6.06.1LTS x64 2.6
Host 208MB/s read, 140MB/s Write (used iometer 2006)
2000 Guest:
Read: 68MB/s
Write: 30MB/s
Tried CentOS 5 x64, CentOS 4.5 x32 (Note, saw 8GB), CentOS 5 x32 (4GB) also same.
Finally tested internal drive, has a better ratio to the raw disk performance:
Internal Raptor drive 80MB/s read and write (sustained)
2000 Guest under CentOS 5 x32 with internal drive
Read: Avg 43MB/s (Stable)
Write: 0-45 Avg 38MB/s Same bad performance
Troubleshooting sucesses and failures:
Noticed Cyclic relationship watching iostat -m 1 with iometer running in the guest, the transfer rate would go up in the guest, while host would stay low, then the guest would drop way down and host would flush it out to disk (with a notably higher rate 40 for guest vs 60 for host), then the cycle would start over. 5-6 second cycle at best after some tweaking. I played around with tweaking /proc/sys/vm/dirty_writeback_centisecs and setting it to 60 or with best result setting it to 0 (tested both on array and internal drive) It served to fix that 14 second delay (now only a second or two at worst), but it was still cyclic. Played with other settings in /proc/sys/vm to no major effect.
Messed with hdparm on the block devices, helped with read performance on host, minimal effect on guest (my standard is hdparm -a 8192 /dev/sxx) 16384 hurt performance.
Most host os tests were done (with the exception of a few iometer tests) with dd if=/dev/zero of=somefileonarraytotest bs=1M count=20000 (or 12000) always returned 230-260MB/s because of the large ram anything less than 8GB would return up to 2.5GB/s
MemTrimRate=0
sched.mem.pshare.enable = "FALSE"
mainMem.useNamedFile = "FALSE"
Tried above with no major effect on disk performance (actually seemed to slow interface down a bit perhaps)
Tried using latest driver from adaptec (aacraid), no effect, still bad.
Was initially doing tests with XFS, but changed to ext3 to use CentOS versions. Made no difference either way though, host performance and guest performance still same.
My conclusion (but not solution) is that because of memory mapping or buffering or both (or ???) that there is a buffer that is being allowed to fill up, then drain while the guest waits for it to drain, then start over again. The bad read performance is very curious on the newer kernels, WHY!
Most of all WHY IS WINDOWS PERFORMING SO MUCH BETTER. I can understand a moderate difference like if the guest had 150MB/s reads and 100MB/s writes, but not 0-40MB/s writes !!![]()
Also note that using noatime on a partition caused VM Server Console not to be able to create a new virtual disk !?!?? (Under CentOS 4.4) Turned off and was fine. Also suspend/resume from disk on shutdown/startup did not work under CentOS 4.4, worked fine elsewhere.
There were lots more things I tried that I can't remember/didn't write down. Scoured the net for answers and this is all I could find.
Need your help.
(Learned a lot about linux over the last several weeks, from boot problems to all kinds of stuff, regardless almost all these versions of linux have installed relatively painlessly on this brand new latest hardware and the OSes have been very stable)