Logging, Monitoring, and Detection

CAP, ACID vs BASE, latency numbers, back-of-envelope estimation, single points of failure — the vocabulary every system designer thinks in.

Core Philosophy: An attack you cannot see is an attack you cannot stop. Page 4.5 established that defenders must watch continuously; this page is the how. Detection rests on a simple, unforgiving chain: systems must record what happens (logging), that record must be collected and watched (monitoring), and meaningful signs of attack must be picked out of it (detection). Break any link and the attacker operates invisibly. This is where your offensive knowledge becomes a defensive superpower — you can detect what you know how to do.

Part 1: The Problem

Across Phases 2 and 3, did you consider what traces your attacks left? Your scans, brute-force attempts, exploitation, privilege escalation — every one caused activity on the target. If nobody records that activity, and nobody examines the recording, the attacker is invisible — and an invisible attacker has unlimited time to do unlimited damage.

A sobering real-world fact frames this page: attackers frequently remain undetected in breached environments for a long time — not because they are invisible by nature, but because the organization was not logging the right things, was not watching the logs, or could not pick the attack out of the noise.

Detection closes this gap. It rests on three links — logging, monitoring, detection — and here, more than anywhere, your offensive training pays off: you cannot write good detection for an attack you do not understand, and you understand these attacks because you performed them.

Part 2: The Concept — The Detection Chain

Detection is a chain, only as strong as its weakest link:

   LOGGING        →    MONITORING      →    DETECTION
   systems record      logs collected       meaningful signs
   what happens        & continuously       of attack identified
                       examined             within the data
   ─────────────────────────────────────────────────────────
   no logging =        logs nobody          data nobody can
   nothing to see      reads = blind        interpret = noise

• Logging — systems and applications must record events. Raw material. No log, no evidence, no detection. (Configured during hardening, 4.4.)
• Monitoring — logs must be collected centrally and continuously examined. Logs unread on a thousand machines are nearly useless.
• Detection — within all that data, the meaningful signs of attack must be identified and surfaced as alerts — the hard, skilled part.

Part 3: Logging — Recording What Happens

The principle: systems, applications, and network devices must record security-relevant events, so there is a trustworthy record of what happened.

What should be logged:

• Authentication events — logins, logouts, and especially failed logins (failed-login patterns reveal brute force and credential stuffing — 2.6).
• Access events — what was accessed, by whom; access-control failures especially (repeated “denied” events signal probing — 2.7, 4.2).
• System changes — configuration, account, software, and privilege changes (privilege changes can signal escalation — 3.4).
• Network activity — connections, especially unusual ones (scanning and lateral movement leave traces — 3.1, 3.5).
• Application activity — significant application events and errors.

Doing logging well matters as much as doing it:

• Log enough, but meaningfully. Too little misses the attack; too much buries it. Log security-relevant events with enough detail to investigate.
• Never log sensitive data. Do not write passwords, full payment data, or secrets into logs — logs themselves become a target (recall 2.9’s exposed log files).
• Protect log integrity. An attacker who can alter or delete logs erases their own traces. Logs should be sent somewhere central and tamper-resistant — another reason centralized collection (Part 4) matters.
• Synchronize time. Logs from different systems must share a consistent time source, or you cannot correlate events into a coherent timeline.

Part 4: Monitoring — Collecting and Watching, and the SIEM

Logs scattered across many systems are nearly useless — an attack spans multiple systems, and its traces are fragments in many places. Monitoring means centralizing those logs and continuously examining them.

The central tool, introduced in 4.5, is the SIEM (Security Information and Event Management) system. What it does:

• Aggregation — collects logs and security data from across the whole environment into one place.
• Normalization — puts differently-formatted logs into a consistent form so they can be analyzed together.
• Correlation — the key capability. It connects related events across different systems — a failed login here, an access-control failure there, an unusual connection elsewhere — into a picture. Individually each event is nothing; correlated, they reveal an attack. (This is the detection-side mirror of chaining from 2.11: attackers chain steps; defenders correlate the traces of those steps.)
• Alerting — generates alerts when something matching a detection rule (Part 5) occurs.
• A search/investigation interface — the analyst’s workspace for investigating (4.5).

Modern environments also use other monitoring — endpoint detection on individual machines, network monitoring — but the SIEM is the hub that brings it together.

Part 5: Detection — Finding the Attack in the Data

The hardest, most skilled link: identifying the meaningful signs of an attack within all the collected data, and surfacing them.

Detection rules. Most detection runs on rules — defined logic describing what an attack looks like, which generate an alert when matched. Two complementary kinds:

• Signature/known-pattern detection — looks for known attack patterns and known-bad indicators (a known-malicious address, a known attack signature). Reliable for known threats; blind to genuinely novel ones.
• Anomaly/behavior detection — looks for deviations from normal — a login at an unusual time, a user accessing far more than usual, traffic to an unexpected place. Can catch novel attacks; produces more false positives.

Indicators of compromise (IOCs) — specific observable signs that an attack has occurred or is occurring (a known-bad address, a suspicious file, a particular pattern of activity). Detection partly hunts for IOCs; threat intelligence (4.5) supplies known ones.

Detecting your Phase 2–3 attacks — the purple-team payoff. This is where your offensive training becomes defensive power. For every attack you learned, you can now reason about its detection signal:

• A port scan (3.1) → many connection attempts across many ports from one source in a short time.
• Brute force / credential stuffing (2.6) → a burst of failed logins, or failed logins across many accounts.
• Exploitation (3.3) → unusual process activity, unexpected new connections, a service behaving abnormally.
• Privilege escalation (3.4) → unexpected privilege changes, a low-privileged account suddenly doing privileged things.
• Lateral movement (3.5) → unusual internal connections, one account authenticating to many machines.
• Web attacks (Phase 2) → injection-like payloads in inputs, access-control failures, anomalous request patterns.

You can detect what you understand — and you understand all of this, because you did it.

The false-positive challenge. The eternal tension of detection: rules too broad flood analysts with false positives (driving the alert fatigue of 4.5); rules too narrow miss real attacks (false negatives). Good detection is tuned continuously toward catching real attacks while keeping noise manageable — never “set and forget.”

Part 6: Detection as a Continuous, Improving Practice — and What’s Next

Detection is never finished. Attackers change techniques; environments change; rules drift out of date; false positives need taming. Detection is a continuously improving practice — new detections written for new techniques, existing rules tuned, gaps closed as incidents reveal them. This is the SOC’s improvement loop (4.5) made concrete, and it embodies the purple team idea (1.5): every incident, and every offensive exercise, teaches the defenders what to detect better.

How it connects:

• It depends on hardening (4.4) — logging is configured there; detection needs those logs.
• It is the SOC’s core mechanic (4.5) — this page is how the SOC’s “detect” step works.
• It feeds incident response (4.7) — detection is what triggers response; you cannot respond to what you never detected.
• It is sharpened by offense (Phases 2–3) — you detect what you understand. The clearest demonstration of why this curriculum teaches both halves.
• It connects to Phase 6 — AI is increasingly applied to detection (handling scale, spotting anomalies), examined in 6.7; and AI systems themselves need monitoring (6.5).

🔑 The deep lesson: detection turns “assume breach” from a slogan into a capability. You accept attackers will get in (1.1) — so you ensure that when they do, they are seen: systems log their traces, monitoring collects those traces centrally, and detection picks the attack out of the noise. And the better you understand attacks — which, after Phases 2 and 3, you do — the better you can detect them.

📓 Key Terms

🧪 Hands-On Lab

Generate attack traffic only against your own lab (Phase 0.5). These tasks reuse your Phase 2–3 attacks against your own systems, then detect them.

Task 1 — Configure logging. On a lab VM, ensure security-relevant logging is enabled — authentication events especially. Locate the logs (recall /var/log from 0.2).

Task 2 — Generate and find attack traces. Attack your own lab VM with something from Phase 3 — a brute-force attempt against a service (3.2), or a port scan (3.1). Then examine the logs and find your own attack’s traces. Use the piping skills from 0.2 (e.g. filtering an auth log for failed logins).

Task 3 — Explore a SIEM. Set up or use a free/community SIEM or log-analysis platform. Feed it logs from your lab. Explore aggregation, search, and correlation. Understand the analyst’s working environment.

Task 4 — Write a detection rule. In your SIEM/log tool, write a simple detection rule for an attack you performed — e.g. “alert on many failed logins from one source in a short window” (brute force). Trigger it by re-running the attack. You just built detection for an attack you understand.

Task 5 — Map attacks to signals. Take five attacks from Phases 2–3. For each, write the detection signal it produces and a rough detection-rule idea. This is the purple-team skill — and genuinely portfolio-worthy (Phase 7).

Task 6 — Experience the false-positive tension. Make your Task 4 rule deliberately too broad and see it fire on benign activity; then tighten it. Feel the tuning trade-off between false positives and false negatives.

Task 7 — Write a detection note. In Notion, create “Detection & Monitoring” — the detection chain, what to log, what a SIEM does, and your attack-to-signal mapping. Reference material for blue-team work.

⚠️ Common Mistakes

• Not logging, or logging the wrong things. No log, no detection. Log security-relevant events meaningfully — and never log secrets.
• Logging but never watching. Logs nobody examines are nearly useless. Centralize (SIEM) and continuously monitor.
• Not protecting log integrity. An attacker who alters or deletes logs erases their traces. Send logs to a central, tamper-resistant place.
• “Set and forget” detection. Attackers and environments change; rules drift. Detection is continuously tuned and improved.
• Tuning rules too broad or too narrow. Too broad floods analysts (alert fatigue); too narrow misses attacks. Tune toward catching real attacks with manageable noise.
• Forgetting the offense link. You detect what you understand. Your Phase 2–3 knowledge is exactly what lets you write good detection.

✅ Recap & What’s Next

• Detection rests on a chain — logging (record events), monitoring (centrally collect and watch, via a SIEM that correlates across systems), detection (identify attacks in the data via rules, signatures, and anomaly analysis).
• It is sharpened by offense: you detect what you understand, and after Phases 2–3 you understand these attacks — you can map each to its detection signal.
• It is a continuously improving practice balancing false positives against false negatives, and it turns “assume breach” into a real capability.

Next (4.7): Detection tells you an attack is happening. Page 4.7 is what you do next — incident response: the structured process of containing, investigating, eradicating, and recovering from a security incident.