Virtual Machines¶
- Problem with secure OS:
- Often define a new API
- Application must be ported -> expensive
- Alternative: Run conventional OS inside a VM
- Approach with OS security:
- Protection domain on one physical platform
- Purpose: Improve resource utilization
- Idea:
- Use Virtualization
- Use system software for isolation (instead of sharing)
- Isolation can result in security
- Definition Virtualization:
- a technique for hiding the physical characteristics of computing resources from the way in which other systems applications and users interact with those resources
- Definition Virtual machine:
- Single physical resource can appear as multiple logical resources
- Classification:
- Type 1 (native)
- Lowest layer of software is VMM
- e.g., Xen, VAX
- Type 2 (hosted)
- Runs on a host operating system
- e.g., JavaVM, VMWare
- Trust host OS
- Type 1 (native)
- Separation kernel:
- Type of security kernel
- Kernel simulating a distributed environment
- e.g., VAX VMM, seL4
- Approaches:
- VAX (isolate complete OS)
- Dune (containers)
- Proxos (isolate small trusted OS)
Hardware virtualization¶
- What can be virtualized?
- CPU virtualization
- Instructions (sensitive instruction must be mediated by the VMM)
- Memory virtualization
- MMU
- Nested/extended page tables
- I/O virtualization
- IOMMU (requires chipset support: configuration for address translation)
VMM as Reference Monitor¶
- Question: Can a VMM achieve the reference monitor guarantees?
- Complete Mediation:
- Mediate all interaction of VM and hardware
- Tamperproof:
- Hypervisor is protection between the VMs
- Verification:
- Simple
- Policy and correct code required
VAX¶
- Virtual Address eXtension (by DEC)
- Karger et al., 1991
- Goals
- A1-assured VMM system (Trusted Computer System Evaluation Criteria, TCSEC)
- Mediation: VM (subject), volume (object)
- Tamperproof: Minimal TCP (VMM only)
- Verification:
- Code assurance
- Policy assurance
- Design tasks:
- Virtualize processes
- Security-sensitive instructions should be privileged
- Privileged instruction: Access kernel memory
- Ring usage
- Four rings already used
- Compress guest OS kernel and executive into single ring
- I/O emulation
- Access to I/O must be emulated
- Access to devices from guest OS is not possible due to ring 1
- Self-virtualizable (recursive VMM/VM)
- Virtualize processes
Architecture¶
- Inspired by Multics: Layered architecture Multics is layered???
- Protection state:
- Subjects: VMs are labelled
- Objects: Devices/storage

Policy¶
- Multilevel Security (secrecy)
- Biba (integrity)
- Privileges
- Exceptional accesses, which can’t be modeled by MLS/Biba
- e.g., access to higher-privilege resource,
SECUREcall for management - Audit privileges
Evaluation¶
- Complete mediation
- All operations:
- Security-sensitive instructions in the processors instruction set must be privileged
- Virtualize processes
- I/O emulation
- All resources:
- Devices are not accessible by VM
- Device access is mediated by VMM
- Verify mediation: A1-assurance
- All operations:
- Tamperproof:
- Protection of VMM:
- Code runs in ring 0
- Small administration using the
SECUREcommand - VAX does not filter input data
- Protection of TCB:
- No trusted code outside of ring 0
- Only trusted path for admin (
SECUREcall)
- Protection of VMM:
- Verification:
- Code: A1-assurance
- Policy: MLS + Biba express security goals
Challenges¶
- A1-assurance does not capture everything
- Open issues:
- Device driver management
- No network support Is this true???
- Lots of assembler code
- Covert channels countermeasures
- Implications of privileges Which???
- Interesting mechanisms:
- Pioneer system for using virtualization for security
- Architecture
Modern VM¶
- Examples: VMware, Xen, KVM (kernel-based virtual machine)
- Open problem: Inter-VM communication
- e.g., Network communication, VM-VMM communication (hypercalls)
- VAX had no network support Is this true???
- This communication needs to be controlled
- Industry approaches:
- IBM sHype: Reference monitor for controlling network
- VMware NetTop: Add-on for VMWare, VMs of same label may communicate
- Xen Security Modules:
- Reference validation of Xen hypervisor
- Domain 0: SELinux (manage hardware access)
- How do they work???
- Resources for communication:
- Memory
- Devices
- IPC
- VMs themselves (e.g., VMCI, IPC between VMs)
- Container systems:
- Virtualization on Application Binary Interface
- Linux-container runs Linux-like system on Linux host OS
- Resource controlling:
cgroups - Each processor gets own namespaces
- Container is isolated
- Task: Configure access control for containers
- e.g., SELinux as host and guest
- e.g., Ad-hoc solutions via MAC
- Comparison container/VM:
- Containers share OS (bad)
- Container have one application (better)
- Container have limited attack surface (because isolated)
- OS vs VM:
- OS: Broken often
- VM: Coarse-grained control using isolation
- Highly-application dependent
Dune¶
- Goal: Safe access to hardware features from processes
- Benefit if access is directly possible
- e.g., performance
- Problem: Providing direct access and maintaining security is a problem (i.e., access to kernel)
- Idea:
- Virtualize process instead of hardware
- i.e., Container system
- Alternative: VM with custom OS for special hardware features
- Dune module
- Loadable kernel module
- Using Intel VT-x
- Virtualization hardware
- Add two new privilege classes beside the ring model:
- VMX Root operation (for VMM)
- VMX non-root operation (for VMs)
- Process abstraction
- Process can enter Dune mode irreversibly to access hardware features
- Provides fast and safe access to special hardware features
- e.g., virtual memory registers, page tables, interrupts, exception, system calls
- Intel VT-x
- VMM runs in root mode
- VM in non-root operation
- Dune process invokes system call (using library
libDune)
- Architecture
- Kernel: VMX Root mode
- Dune module: Only way to access kernel are syscalls and traps (intercepts VM exists)
libDune:- Maybe to create Dune-aware processes?
- Dune mode: VMX non-root mode
- Memory management
- Performed by Dune module
- Goal: Enable direct access to page tables without access to physical memory
- Dune processes contain a guest-virtual memory like the MMU???
- Protected by the Extended Page Table (EPT)
- EPT is synced with kernel page table
- What is the problem of Dune???
- Does Dune implement a Reference Monitor???

Proxos - Splitting Interfaces¶
- Ta-Min et al. 2006
- Scenario:
- Application needs OS services
- Application doesn’t trust OS
- Approaches
- Run app directly on VMM: No services
- Run services on custom OS: Expensive (must be written)
- Idea:
- Re-use small trusted system enabling the secure use of services (private OS)
- Separate application from other applications and kernel
- Private OS (running sensitive parts)
- Commodity OS
- Does Proxos implement a Reference Monitor???
Architecture¶
- Based on Xen
- Components:
- Private OS (running sensitive parts)
- Commodity OS
- Requires separation of app into sensitive and non-sensitive part
- Programming-model of SGX and TrustVisor
- Sensitive part is placed into private OS
- Non-sensitive part is placed in common OS
- Sensitive part: Communicate with other applications (using syscalls)
- Use policy to decide whether a syscall needs to be performed by the private OS or the commodity OS

Guarantees¶
- Assumptions:
- VMM enforces separation correctly
- Routing rules are correctly specified
- Secrecy and integrity enforces (for sensitive private application)
- Availability not guaranteed
Configuration¶
- Need to specify routing rules (which syscall goes where)
- Important: Consider arguments
- Approach: Partition system calls by resources
- Disk
- Network
- UI
- Randomness (VMM)
- System time (VMM)
- Memory (private OS)
Workflow¶
- Start-up
- Host process enters sensitive area
- Commodity OS creates buffer
- Commodity OS passes buffer address and application name to VMM
- VMM creates the private VM
- Pass buffer to VM
- Associate buffer with private VM ID
- Private VM: Map buffer into address space
- Commodity OS saves private VM ID and host PID (execute incoming calls with host PID)
- Security mechanism: VMM checks if VM ID requested the mapping was the VM created the mapping
- System call forwarding (e.g., ocalls)
- Proxos informs kernel for forwarding system call
- Schedule host process
- Execute system call for host process
- Proxos receives result (after another context switch)
- Example SSH server
- Porting required 35 rules and 108 LOC changed
