Skip to main navigation Skip to search Skip to main content

Using a Hypervisor for Enforcing Control Flow Integrity in Native Compiled Programs

Research output: Chapter in Book/Report/Conference proceedingChapterpeer-review

Abstract

Many attacks change a running program’s control flow. Such attacks include return-oriented programming, return to libc, library injection, etc. These attacks modify the running code and can achieve virtually any behavior while preserving the attacked software execution permissions. Thus, an attacker can run any code with full execution permissions by successfully implementing a control-flow attack on a server. Control-flow inspection examines the control flow of a running process. If an abnormal control flow is detected, its root cause is determined. The root cause may be a software bug or an attack. An automated control-flow inspection mechanism may allow a security analyst to detect an attack promptly before the attacker causes significant damage to the protected system. There are multiple systems for offline (static) control-flow inspection. For example, security analysts can use memory dumps obtained by software such as LiME using a memory analyzer such as Volatility. However, offline introspection is very limited, as obtaining the memory dump for analysis consumes time, which may harm production systems for long periods. Therefore, in practice, only a few samples are taken and analyzed, which may be too little and too late. We propose a hypervisor-based approach for online introspection (dynamic analysis) of the control flow in compiled binaries. Our system can detect Return-Oriented Programming (ROP) and other control-flow hijacking attacks when the source code is unavailable. We present the design of our system and analyze its effectiveness and efficiency.

Original languageEnglish
Title of host publicationStudies in Big Data
PublisherSpringer Science and Business Media Deutschland GmbH
Pages483-501
Number of pages19
DOIs
StatePublished - 2026

Publication series

NameStudies in Big Data
Volume183
ISSN (Print)2197-6503
ISSN (Electronic)2197-6511

Keywords

  • Control flow
  • Hijacking
  • Hypervisor

Fingerprint

Dive into the research topics of 'Using a Hypervisor for Enforcing Control Flow Integrity in Native Compiled Programs'. Together they form a unique fingerprint.

Cite this