Achieving High Performance Engineering in Cloud-Native SOCs: Mastering Contextual Detection

The landscape of cloud-native security is constantly evolving, presenting unique challenges for Security Operations Centers (SOCs). A recent discussion on GitHub Community, initiated by Leonardo-cyber-vale, highlighted a critical roadblock: effectively detecting Living off the Land (LotL) techniques in dynamic, containerized multi-cloud environments. The core issue? A high 'signal-to-noise' ratio when legitimate administrative tools like kubectl or aws-cli are repurposed by adversaries, often making traditional signature-based and even standard behavioral baselining ineffective due to rapid DevOps workflows.

This challenge underscores the need for a sophisticated approach to detection engineering, aiming for high performance engineering in security operations. The community sought insights into contextual enrichment, eBPF management, and alert prioritization.

Contextual Enrichment: A Hybrid Approach to Data Intelligence

One of the central questions revolved around implementing contextual enrichment at scale. Should telemetry be enriched at the ingestion layer or during query time in the SIEM/Data Lake?

Community member Thiago-code-lab advocated for a hybrid approach, strategically splitting enrichment based on data volatility:

• Ingestion-Time (Stream): Ideal for "hard," immutable metadata. Using tools like Kafka, immediate stamping of data points such as Cluster ID, Node, and Container Image Hash ensures that ephemeral IPs can be accurately mapped to specific microservices before they are reused. This is crucial for maintaining data integrity and reducing future correlation pain.
• Query-Time (SIEM/Data Lake): Better suited for volatile context. Complex IAM role metadata, like a user's sensitive group membership at a particular moment, is more efficiently handled during the hunt or query phase in the Data Lake (e.g., S3 + Athena/Trino). This avoids introducing latency bottlenecks into the ingestion pipeline, preserving the flow for critical real-time data.

eBPF and Correlation: Event Generation Over Stateful Management

The discussion also touched upon the overhead of stateful correlation when using eBPF for runtime security tools like Falco or Tetragon, especially when trying to link kernel-level events with high-level cloud audit logs.

Thiago-code-lab's perspective was clear: treat eBPF strictly as an Event Generator, not a correlation engine. Attempting to maintain stateful correlation between kernel syscalls and CloudTrail logs at the agent level introduces excessive overhead and instability. Instead, the recommended strategy involves:

• Streaming raw eBPF syscalls and cloud audit logs into a unified Data Lake zone.
• Utilizing batch jobs (e.g., Airflow) or windowed stream processing to correlate specific identifiers, such as k8s_pod_name from eBPF with userIdentity from CloudTrail. This decouples event generation from complex correlation logic, leading to a more robust and scalable system.

Alerting vs. Data Lake Hunting: Prioritizing Deeper Context

Given the highly dynamic nature of DevOps, the community debated whether to prioritize high-fidelity, low-volume alerts or 'Data Lake' hunting with ML-driven anomaly detection. For LotL techniques, where legitimate administrative actions can easily mimic malicious ones, Thiago-code-lab strongly preferred Data Lake Hunting over real-time alerting.

The rationale is compelling: since legitimate administrators frequently use tools like kubectl, a blocking rule is inherently risky and prone to false positives. Instead, focusing on ML-driven anomaly detection within the Data Lake allows for identifying deviations from typical behavior, such as:

"This user typically runs 5 kubectl exec commands a week, but just ran 50 in one hour."

This approach significantly reduces pager fatigue and provides security analysts with deeper context during investigations, aligning with principles of high performance engineering by optimizing analyst time and reducing noise.

This community insight highlights the evolving strategies required for modern cloud-native SOCs. By embracing hybrid enrichment, treating eBPF as a powerful event source, and shifting towards intelligent data lake hunting, organizations can build more resilient and effective detection pipelines against sophisticated threats.