NHVM: Design and Implementation of Linux Server Virtual Machine Using Hybrid Virtualization Technology

Abstract

The virtualization technology has been developed rapidly with the growth of the hardware supported virtualization technologies and the appearance of the various services. Many researchers are concentrating on developing the virtualization technologies which are recognized as the most significant core technology of the IT applications such as green IT and cloud computing. The major processor companies have introduced various hardware supported virtualization technologies for overcoming the limitations of the software virtualization technologies. Until now, however, the hardware supported virtualization technologies are used only for supporting the full-virtualization technologies. In this paper, we introduced our efficient server virtual machine, which merged the hardware supported virtualization technologies with conventional para-virtualization technologies. Through our proposed hybrid virtualization technology, we could reduce the complexity and overhead of traditional server virtual machines. And lastly, we evaluated and compared our proposed virtual machine with traditional server virtual machines through the experiments.

Full text

Page 1

NHVM: Design and Implementation of Linux Server Virtual Machine
using Hybrid Virtualization Technology

Inhyuk Kim, Taehyoung Kim, Young Ik Eom
School of Information and Communication Engineering
Sungkyunkwan University
Suwon, Korea
{kkojiband, kim15m, yieom}@ece.skku.ac.kr






Abstract— The virtualization technology has been developed
rapidly with the growth of the hardware supported
virtualization technologies and the appearance of the various
services. Many researchers are concentrating on developing
the virtualization technologies which are recognized as the
most significant core technology of the IT applications such as
green IT and cloud computing. The major processor
companies have introduced various hardware supported
virtualization technologies for overcoming the limitations of
the software virtualization technologies. Until now, however,
the hardware supported virtualization technologies are used
only for supporting the full-virtualization technologies. In this
paper, we introduced our efficient server virtual machine,
which merged the hardware supported virtualization
technologies with conventional
technologies. Through our proposed hybrid virtualization
technology, we could reduce the complexity and overhead of
traditional server virtual machines. And lastly, we evaluated
and compared our proposed virtual machine with traditional
server virtual machines through the experiments.
para-virtualization
Keywords-Linux; Server virtual machine; Xen; lguest; Para-
virtualization; Full-virtualization; Hybrid virtualization
I.
INTRODUCTION
The virtualization technologies have been studied for
increasing utilization of mainframe servers since 1960’s. The
main concepts of the virtualization originally came from the
situation that lots of servers had low utilization rate about
10~20% at that time. For enhancement, many servers
operated on the virtualized machine which exists in one real
physical machine to increase server usage as well as
maintenance security. The
technologies have been mainly developed for the mainframe
servers when desktop PC didn’t have enough performance.
The research about the virtualization technology for desktop
PC has been started actively since 1990’s through the
improvement of desktop PC. At present there are exist
several server virtual machines for desktop PC such as Xen
which has been developed and maintained by open source
traditional virtualization
community and VMWare ESX Server has been published by
VMWare Corporation.
Recently, some of the research area in IT have been
focused on the global environment. Researchers have coined
the term ‘Green IT’, and there are a significant number of
cases to increase the utilization of servers through the
virtualization technology in data centers. And the cloud
computing service that is internet-based development and
use of computer technology is also based on the
virtualization technology. For example, Amazon Elastic
Compute Cloud service supports server platform cloud
service using Xen, the one of the most famous open source
server virtual machine. There are lots of other cloud services
with Xen, VMWare solutions, or any others.
The three components of virtualization technologies are
process virtualization, memory virtualization and device
virtualization. The process virtualization supports privileged
instruction handling, interrupt/exception handling, and so on.
And the memory virtualization supports address translation
by paging and memory protection. Last the device
virtualization supports handling the virtual devices of the
guests. There are two types of processor virtualization
technologies according to the necessity of modification of
the guest kernel. The technologies that require modification
of the guest kernel are called as the para-virtualization. On
the other hand, the technologies that don’t require guest
kernel modification are called as the full-virtualization. And
memory virtualization technologies also have two types of
schemes. The shadow paging scheme don’t need to modify
guest kernel. But, the direct paging scheme requires
modification of the guest kernel. Even though both the para-
virtualization and the direct paging scheme have
inconvenient of modification of the guest kernel, they shows
the best performance among the all of the virtualization
technologies. So, the server virtual machines have used the
para-virtualization technology for processor virtualization,
the direct paging scheme for memory virtualization and
frontend/backend driver model for device virtualization.
The most famous server virtual machines, Xen and
VMWare ESX Server are based on the para-virtualization
2010 International Conference of Computational Science and Its Applications2010 International Conference on Computational Science and Its Applications
978-0-7695-3999-7/10 $26.00 © 2010 IEEE
DOI 10.1109/ICCSA.2010.49DOI 10.1109/ICCSA.2010.49
171171978-0-7695-3999-7/10 $26.00 © 2010 IEEE

Page 2

technology. Even though the para-virtualization technology
requires the modification of the guest kernel, it is faster than
others with different virtualization technologies. Because in
the para-virtualization technology, the modified guest kernel
codes that don’t have privileged instructions at all can
execute directly on a physical machine. A few years ago, the
major processor corporations added some instruction sets
into their processor for virtualization. This enables to
develop efficiently the virtual machine and to increase the
performance of guests compared with the virtual machine
using software-only virtualization technologies. So far, the
hardware supported virtualization technology has been used
for the full-virtualization technology which used to
emulation and binary translation scheme, but this research
will introduce an advanced server virtual machine using
hybrid virtualization technology which combines the para-
virtualization technology
virtualization technology.
The remainder of this paper is organized as follows. In
the section 2, we will present related works such as Xen and
lguest. Section 3 will introduce our proposed virtual machine
and section 4 will show the design and implementation
details, and in the next section 5, we will show the evaluation
results through the experiments. Finally, section 6 will give
the conclusion and our plans for future works.
with hardware supported
II.
RELATED WORKS
A. Xen
Xen was released by Cambridge University and has been
developed by XenSource, Inc. At first Xen supported only
linux server virtual machine, but nowadays it also supports
non-modified guest operating system using the hardware
supported virtualization technology. There are lots of
companies such as Amazon, IBM and so on, which use Xen
for commercial sale[1][4]. Fig. 1 represents the architecture
of Xen based on the para-virtualization technology.

DomU
Process
Modified
Linux Kernel (guest)
Process
Dom0
Process
Linux Kernel
(guest)
Process
DomU
Process
Modified
Linux Kernel (guest)
Process
Xen(Hypervisor)
IntelMemoryDisk...
Frontend driverBackend driver
Native driver

Figure 1. The Architecture of Xen.

As you can see, the hypervisor manages physical
resources and provides the services to many guests such as
scheduling, memory management, IPC and so on. The Dom0
guest domain manages the requests from virtual devices
through the frontend driver of guests and applies those
requests to the real device.
The memory virtualization of Xen uses the direct paging
scheme. This scheme allocates all the necessary physical
memory when guest starts, and the hypervisor notifies this
conversion table to the guest. The guest should manage all
the page directories and tables using this table directly.
B. Lguest
lguest was developed by Rusty Russell, and the virtual
machine that is built in the linux kernel using the para-
virtualization technology. It is quite light-weighted version
of the virtual machine which has just only 3,000 lines of
code, so it has several limitations following in server virtual
machine, but this is one of the best resource for research[5].
Fig. 2 shows the architecture of lguest.

Guest
Modified
Linux Kernel (guest)
lguestprocessGuest
Modified
Linux Kernel (guest)
Linux Kernel (hypervisor)
IntelMemoryDisk...
Frontend driverBackend driver
Native driverlguestmodule
virtio
ProcessProcessProcessProcess

Figure 2. The Architecture of lguest.

As Fig. 2 shows, lguest consists of the hypervisor as
built-in module and process. This hypervisor as supports
memory management, IPC, etc. as like Xen hypervisor, and
lguest process supports virtual device management using
virtio.
The lguest modifies the linux kernel for guest using
paravirt_ops which helps the modification of the guest kernel
easier for the para-virtualization technology. Furthermore,
just one compiled kernel image can be used for host and
guest simultaneously. For example, write_cr3() that is used
for changing page directory changes directly processor’s
CR3 register in the case of host, but in the case of guest
should request to host using hypercall. For this, the linux
kernel should verify the system type that is received while
booting, and if the system type is guest, write_cr3() should
be replaced by paravirt_ops.
The device virtualization uses frontend/backend driver
model using virtio which supports ring buffer mechanism
and virtual interrupt management. At this time, lguest
supports virtual console, block device and network device.
These are easily developed with virtio[6].
III.
NHVM ARCHITECTURE
This paper introduces the advanced virtualization
technology merged the para-virtualization technology with
hardware supported virtualization technologies. The old-
fashioned server virtual machines used the para-
virtualization technology for processor virtualization and the
direct paging scheme for memory virtualization as well as
frontend/backend driver model for device virtualization.
172 172

Page 3

These virtual machines showed the best performance among
the virtual machines using the software-only virtualization
technologies, and these are used by the several server virtual
machines in commercial area. However, the hardware
supported virtualization technologies were released and
several desktop virtual machines have used it efficiently
since a few years ago, but there is no case of a server virtual
machine with hardware supported virtualization technology.

Guest
Modified
Linux Kernel (guest)
NHVM process Guest
Modified
Linux Kernel (guest)
Linux Kernel (hypervisor)
Intel MemoryDisk ...
Frontend driver Backend driver
Native driver
NHVM hypervisor
ProcessProcessProcessProcess
VMCS
VT-x
vmexit handler
VMCS
virtio

Figure 3. The Architecture of NHVM.

Fig. 3 shows the architecture of Next-generation Hybrid
Virtual Machine, our proposed virtual machine. We applied
Intel VT-x technology for enhancing host-guest context
switching and interrupt/exception handling, the shadow
paging scheme and frontend/backend driver model into our
virtual machine. We used paravirt_ops for modifying the
guest linux kernel, so we can use same kernel image both
host and guest kernel. For frontend/backend driver model,
we used virtio that is the communication between frontend
driver of the guest kernel and backend driver of the NHVM
process, and the NHVM process communicates with the
native drivers through the Virtual Filesystem of Linux
Kernel like general processes.
IV. DESIGN AND IMPLEMENTATION
In this section, we will introduce the three of the most
important technical issues of NHVM. There are host-guest
context switching and interrupt/exception handling as well as
shadow paging.
A. Host-guest context switching
Host-guest context switching is occurred many times for
guest running. When the host needs to handle the hardware
interrupt, or the guest should change the page directory and
kernel mode stack for context-switching, and the guest use
the virtual devices, the host-guest context switching will be
occurred. Therefore, improving the performance of host-
guest context switching is one of the important issues.
NHVM uses Intel VT-x for faster host-guest context
switching. So, we don’t need to share host’s and guest’s
virtual address spaces like Xen and lguest. Fig. 4 shows the
basic scenario.

Host
Linux Kernel
User Process
VMCS
Guest
Linux Kernel
User Process
1. VMRESUME
3. VMEXIT
2. Interrupt/Exception

Figure 4. The host-guest context switching process of NHVM.

As you can see, NHVM allocates each VMCS(Virtual
Machine Control Structure) to each guest for Intel VT-x. The
VMCS is the structure which has lots of information for
guest running and is managed directly by processor. For
entering the guest, we use VMLAUNCH or VMRESUME
instruction. These instructions save the host context
information to VMCS and restore the guest context
information from VMCS. When the VMEXIT is occurred
for the host handling, the processor saves the guest context
information to VMCS and restore the host information from
VMCS oppositely. After then, the host handles proper
VMEXIT exception, the guest can rerun. As we explained so
far, the host-guest context switching with Intel VT-x is
almost handled by the processor, so it’s fast and easy to
implement[2][3].
B. Interrupt/Exception handling
All of the hardware interrupts are handled firstly by the
host in virtual machine by using the para-virtualization
technology. If an interrupt is occurred while the guest
running, the interrupt should be handled immediately after
switching to the host. And if needed, the host can redirect
that the interrupt to the guest as virtual interrupt.
In NHVM, the host linux kernel manages all of the
physical devices, so the host should handle all of the
interrupts. When the interrupt is occurred during the guest
running, VMEXIT exception will be occurred immediately
by Intel VT-x and after switching to the host, the interrupt
handler of the host kernel will be called. If the host needs to
redirect some interrupt to the guest, the host should set the
virtual interrupt on the guest’s VMCS as soon as the guest
receives that. As explained so far, interrupt/exception
handling using Intel VT-x,
interrupt/exception handling in non-virtualized linux kernel.
and it’s similar to
C. Shadow Paging
In the case of the para-virtualization technology, there are
various kinds of the direct paging schemes like shadow
paging scheme and hardware supported paging scheme for
memory virtualization. However, the direct paging scheme
shows the best performance and has easier implementation
way, it’s almost used only for server virtual machines.
NHVM uses the shadow paging scheme which is used by
the full-virtualization technology. The guest maintains the
page directories and tables using given virtual physical
173 173

Page 4

address, but actually the shadow page directories and tables
which are processed by the host is used by processor. Fig. 5
shows the basic scenario of the shadow paging scheme.

Shadow paging
PGD
PTE
PGD
PTE
Guest
Host
Address
access
Synchronization

Figure 5. The shadow paging scheme.

As Fig. 5 shows, the virtual address of the guest is
translated by the shadow page tables. For the translating
process, the guest’s page tables and the shadow page tables
should be synchronized accurately, so the guest’s page fault
exception should be cared by the host having a higher
priority. If there is no page frame in the guest’s page tables,
the host should redirect the page fault exception to the guest,
and the guest should handle that. If there is a page frame in
the guest’s page tables, the host should update the shadow
page tables and rerun the guest without redirecting page fault
exception to the guest. What is more, the guest can modify
the shadow page tables directly by using paravirt_ops. This
is more easy way to implement the shadow paging scheme
than through the full-virtualization technology, but set_pte()
which don’t have virtual address argument can’t be handled
efficiently.
V.
EVALUATION
In this section, we compared and evaluated with Xen and
lguest through several experiments.
A. Virtbench
Virtbench is micro-benchmark tool for server virtual
machine by Rusty Russell who is the developer of lguest.
This benchmark evaluated native linux, Xen, lguest and
NHVM using virtbench, and Fig. 6 shows the result.


Figure 6. The results of Virtbench.

The result consists of next six tests, and the height of the
vertical axis shows the time required in each test compared
with native linux.

•
One context-switching via pipe
•
One copy-on-write fault
•
Exec client
•
Fork/exit/wait
•
One int-0x80 syscall
•
One syscall via libc

NHVM, which uses the fast host-guest context switching
and the interrupt/exception handling by using Intel VT-x
shows better performance than lguest which uses the
memory virtualization and device virtualization as NHVM.
Nevertheless, Xen shows better performance than NHVM in
the test which evaluates the performance in the reading the
virtual block device through the filesystem, because Xen has
developed better and better frontend/backend driver model.
On the contrary, in the test of context switching using pipe
NHVM shows two times better performance than Xen. This
effect caused by the fast host-guest context switching and the
interrupt/exception handling of NHVM, so we could think
NHVM showed better performance than Xen compare with
other experiments which don’t use virtual devices[7][8].
B. Appbench
This experiments used next four programs, and the
results are shown in the Fig. 7.

•
gcc –c foo.c
•
md5sum linux-2.6.30.tar.bz2
•
gzip -9 initrd
•
kernel build


Figure 7. The results of Appbench.

In the case of md5sum and gzip which required extreme
data processing time and needed a lot of data reading from
virtual block device, showed nearly similar performance in
all the system. In the case of compiling just one simple
174 174

Page 5

source file with gcc, Xen is better than others, because Xen
has efficiently optimized device virtualization. But, the result
of compiling the linux kernel showed NHVM is better than
Xen. The important reason that we’ve got this satisfiable
result is the time for compiling all the files is much less than
the time for reading files via virtual block device. Through
these experiments, we could confirm NHVM has better
efficiency than others on the general CPU bound
process[8][9].
VI. CONCLUSION
This paper introduced brand-new hybrid virtualization
technology using various software and hardware supported
virtualization technologies for enhancing the performance of
server virtual machine. We applied the hardware supported
virtualization technology used for the full-virtualization
technology into the para-virtualization technology, so we
could solve the complexity and overhead problems which
exist in the traditional para-virtualization technology, and
also proved these facts through the experiments.
In the future, we will replace the shadow paging scheme
by hardware supported memory virtualization, and improve
the device virtualization using variable hardware and
software virtualization technologies. And we are going to
study faster and efficient server virtualization technology.


REFERENCES

[1] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R.
Neugebauer, I. Pratt, and A. Warfield, “Xen and the Art of
Virtualization,” Proceedings of 9th ACM Symposium on Operating
Systems Principles, 2003, pp. 164–174.
[2] G. Neiger, A. Santoni, F. Leung, D. Rodgers, and R. Uhlig, “Intel
Virtualization Technology: Hardware Support for Efficient Processor
Virtualization,” Intel Technology Journal, Vol. 10-3, 2006
[3] D. Abramson, J. Jackson, S. Muthrasanallur, G. Neiger, G. Regnier, R.
Sankaran, I. Schoinas, R. Uhlig, B. Vembu, and J. Wiegert, “Intel
Virtualization Technology for Directed I/O,” Intel Technology
Journal, Vol. 10-3, 2006.
[4] Open source Xen. http://www.xensource.com.
[5] R. Russell, “lguest: Implementing the Little Linux Hypervisor,”
Proceedings of the Ottawa Linux Symposium, 2007, pp. 173–177.
[6] R. Russell, “Virtio: Towards a de-facto Standard for Virtual I/O
Devices,” ACM SIGOPS Operating Systems Review, Vol. 42-5, 2007,
pp. 95–103.
[7] K. Adams, and O. Agesen, “A Comparison of Software and Hardware
Techniques for x86 Virtualization,” Proceedings of 12th International
Conference on Architectural Support for Programming Languages
and Operating Systems, 2006, pp. 2–13.
[8] A. Menon, J. R. Santos, and Y. Turner, “Diagnosing Performance:
Overheads in the Xen Virtual Machine Environment,” Proceedings of
the First ACM/USENIX Conference on Virtual Execution
Environments, 2005, pp. 13–23.
[9] L. Cherkasova, and R. Gardner, “Measuring CPU Overhead for I/O
Processing in the Xen Virtual Machine Monitor,” Proceedings of the
Annual Technical Usenix Conference, 2005, pp. 387–390.


175 175