Hi,
I'm implementing an high availability solution based on Remus. My DomU is a Redhat 5.4 64 bits VM, with 8GB of memory allocated.
I'm using kernel linux-2.6.18-xen to run the DomU in paravirt mode, and using DRBD according to the installation recipe. The hardware is a quad core machine with hyperthreading, with 1GB RAM and two cores reserved for Xen (total memory 12GB), ballooning disabled, running Debian Squeeze. Machines are connected by a gigabit ethernet interface, and Remus is configured with 100ms barriers.
I'm experiencing scalability problems inside the VM, with a clear degradation of performance with an increase of the number of cores attributed to it.
I performed a test where I ran dd copying from /dev/zero to /dev/null to simulate CPU load, running as many simultaneous background dd processes as the number of VM cores, and timed the elapsed wall time while changing the number of cores from 1 to 6 - with and without Remus running.
Without Remus running I get the expected result: nearly constant elapsed time. If Remus is active and the machines are in sync, I get parabolic elapsed time growth with ncores>2, rendering the VM nearly unusable. Even with the VM idled, htop shows heavy system usage of most of the available cores.
Did anybody experience similar behavior?
My feeling is that this could be related to the VM kernel having trouble handling the suspend events dispatched through the suspend event channel (which is being used), in a scenario where several cores have to be suspended.
Any thoughts?
Is this a known limitation/problem?
Do you feel using a more recent kernel (2.6.X or 3.1.x from Jeremy Fitzhardinge or the ones maintained by OpenSuse) could circumvent the problem?
Thank you.