Kernel-based Process Monitoring of Network Services

Abstract

Several tracing and intrusion detection techniques have been proposed to capture process behaviour and detect known attack signatures (misuse detection) of derivations from benign behaviour models (anomaly detection) caused by malware or exploitation of benign services. We revisit process tracing techniques and evaluate their tradeoffs for use in anomaly detection systems. We focus on the Linux operating system (OS) which is frequently used for hosting critical network services and limit our discussion to kernel-based approaches as they provide a reasonable tradeoff between isolation and system access compared to user and hypervisor level approaches. We present our extended function tracing architecture based on the new enhanced BPF system and perform a detailed application study of the tracer on wide-spread network services (NTP, DNS and SMTP) including a coverage and behaviour analysis.

We show that services react with request-specific function calls and do not use over 80% of the available system calls. We discuss the implications of our insights for behaviour models and seccomp-based system call filtering. We measure the performance impact of our tracer on the tracee with the industry-standardized SPEC CPU 2006 benchmark revealing an average overhead of 6.67µs per user function call. Finally, we discuss the reaction time of our monitor for live deployment and show that it is best under constantly high event load with a minimum of about 1.9µs per event.

Recommended citation: Schwarz, Fabian Frank, "Kernel-based Process Monitoring of Network Services". Master Thesis. Saarland University. 2017