TY - CHAP
T1 - Using a Hypervisor for Enforcing Control Flow Integrity in Native Compiled Programs
AU - Zaidenberg, Nezer
AU - Kiperberg, Michael
AU - Menachi, Eliav
N1 - Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.
PY - 2026
Y1 - 2026
N2 - Many attacks change a running program’s control flow. Such attacks include return-oriented programming, return to libc, library injection, etc. These attacks modify the running code and can achieve virtually any behavior while preserving the attacked software execution permissions. Thus, an attacker can run any code with full execution permissions by successfully implementing a control-flow attack on a server. Control-flow inspection examines the control flow of a running process. If an abnormal control flow is detected, its root cause is determined. The root cause may be a software bug or an attack. An automated control-flow inspection mechanism may allow a security analyst to detect an attack promptly before the attacker causes significant damage to the protected system. There are multiple systems for offline (static) control-flow inspection. For example, security analysts can use memory dumps obtained by software such as LiME using a memory analyzer such as Volatility. However, offline introspection is very limited, as obtaining the memory dump for analysis consumes time, which may harm production systems for long periods. Therefore, in practice, only a few samples are taken and analyzed, which may be too little and too late. We propose a hypervisor-based approach for online introspection (dynamic analysis) of the control flow in compiled binaries. Our system can detect Return-Oriented Programming (ROP) and other control-flow hijacking attacks when the source code is unavailable. We present the design of our system and analyze its effectiveness and efficiency.
AB - Many attacks change a running program’s control flow. Such attacks include return-oriented programming, return to libc, library injection, etc. These attacks modify the running code and can achieve virtually any behavior while preserving the attacked software execution permissions. Thus, an attacker can run any code with full execution permissions by successfully implementing a control-flow attack on a server. Control-flow inspection examines the control flow of a running process. If an abnormal control flow is detected, its root cause is determined. The root cause may be a software bug or an attack. An automated control-flow inspection mechanism may allow a security analyst to detect an attack promptly before the attacker causes significant damage to the protected system. There are multiple systems for offline (static) control-flow inspection. For example, security analysts can use memory dumps obtained by software such as LiME using a memory analyzer such as Volatility. However, offline introspection is very limited, as obtaining the memory dump for analysis consumes time, which may harm production systems for long periods. Therefore, in practice, only a few samples are taken and analyzed, which may be too little and too late. We propose a hypervisor-based approach for online introspection (dynamic analysis) of the control flow in compiled binaries. Our system can detect Return-Oriented Programming (ROP) and other control-flow hijacking attacks when the source code is unavailable. We present the design of our system and analyze its effectiveness and efficiency.
KW - Control flow
KW - Hijacking
KW - Hypervisor
UR - https://www.scopus.com/pages/publications/105027889366
U2 - 10.1007/978-3-032-08890-1_19
DO - 10.1007/978-3-032-08890-1_19
M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.chapter???
AN - SCOPUS:105027889366
T3 - Studies in Big Data
SP - 483
EP - 501
BT - Studies in Big Data
PB - Springer Science and Business Media Deutschland GmbH
ER -