Links

KVM and Qemu:

Kernel-based Virtual Machine Technology
- https://www.fujitsu.com/global/documents/about/resources/publications/fstj/archives/vol47-3/paper18.pdf
QEMU/KVM VM Execution Basics
Qemu Internals:
- http://itanium.iis.sinica.edu.tw/~chenwj/qemu-tech.html
- http://blog.vmsplice.net/2011/03/qemu-internals-big-picture-overview.html
QEMU Internal: Memory Region, Address Space and QEMU IO
- https://dangokyo.me/2018/03/27/qemu-internal-memory-region-address-space-and-qemu-io/
QEMU Internals: Overall architecture and threading model
- http://blog.vmsplice.net/2011/03/qemu-internals-overall-architecture-and.html
Minimal code for guest execution (check kvm_cpu_exec function of qemu/accel/kvm/kvm-all.c for Qemu implementation and KVM doc for IOCTLs)

open("/dev/kvm")

ioctl(KVM_CREATE_VM)

ioctl(KVM_CREATE_VCPU)

for (;;) {

     ioctl(KVM_RUN)

     switch (exit_reason) {

     case KVM_EXIT_IO:  /* ... */

     case KVM_EXIT_HLT: /* ... */

kvm_arch_vcpu_ioctl_run
- https://elixir.bootlin.com/linux/latest/source/arch/x86/kvm/x86.c#L7700

multiple-iothreads.txt
- https://github.com/qemu/qemu/blob/master/docs/devel/multiple-iothreads.txt

Perf on KVM

Context resolution is a challenge
- https://stackoverflow.com/questions/29847170/how-to-profile-an-app-running-inside-kvm-guest
- https://stackoverflow.com/questions/35997809/how-can-a-specific-application-be-monitored-by-perf-inside-the-kvm
- "Kvm guest may also be non-linux and non-unix"
PEBS for guest OS is disabled
- https://patchwork.kernel.org/patch/8495381
How to measure VM events using perf-event
- https://www.spinics.net/lists/linux-perf-users/msg02268.html

I/O Emulation in KVM/QEMU hypervisor

Understanding Context in VT-x

Figure 6.7 schematically illustrates the aforementioned interactions for the KVM/QEMUhypervisor. Generally, every hosted virtual machine is encapsulated within a QEMU process,such that different VMs reside in different processes. Internally, a QEMU process represents theVCPUs (virtual cores) of its VM using different threads. Additionally, for every virtual device thatQEMU hands to its VM, it spawns another thread, denoted as “I/O thread”. VCPU threads havetwo execution contexts: one for the guest VM and one for the host QEMU. The role of the hostVCPU context is to handle exits of the guest VCPU context. The role of the I/O thread is to handleasynchronous activity related to the corresponding virtual device, which is not synchronouslyinitiated by guest VCPU contexts. For example, handling incoming network packets.
The illustration in Figure 6.7 depicts a VM that has two VCPUs and one I/O device. Theguest VM device driver issues MMIOs/PIOs to drive the device. But the device is virtual, sothese operations are directed at ordinary, read/write-protected memory locations, triggering exitsthat suspend the VM VCPU context and invoke KVM. The latter relays the events back to thevery same VCPU thread, but to its host, rather than guest, execution context. QEMU’s deviceemulation layer then processes these events, using the physical resources of the system, typicallythrough regular system calls.

Publications on Performance Profiling on VM

Du, Jiaqing, Nipun Sehrawat, and Willy Zwaenepoel. "Performance profiling in a virtualized environment." Proceedings of the 2nd USENIX Workshop on Hot Topics in Cloud Computing. No. EPFL-CONF-149112. 2010.
Bratanov, Stanislav, Roman Belenov, and Nikita Manovich. "Virtual machines: a whole new world for performance analysis." ACM SIGOPS Operating Systems Review 43.2 (2009): 46-55.
Nikolaev, Ruslan, and Godmar Back. "Perfctr-xen: a framework for performance counter virtualization." Acm Sigplan Notices. Vol. 46. No. 7. ACM, 2011.
James Snee, Lucian Carata, Oliver R. A. Chick, Ripduman Sohan, Ramsey M. Faragher, Andrew Rice, and Andy Hopper. 2015. Soroban: attributing latency in virtualized environments. In Proceedings of the 7th USENIX Conference on Hot Topics in Cloud Computing (HotCloud'15). USENIX Association, Berkeley, CA, USA, 11-11.

Supported guests by KVM

https://www.linux-kvm.org/page/Guest_Support_Status

Random Reading

The Performance Cost of Shadow Stacks and Stack Canaries