Free through lesson 3

Logging and Monitoring

From what to record, timestamps and formats, through Linux, Windows and cloud logs, central collection and tamper protection, to dashboards and alerts. In 30 lessons you learn to design logs that let you trace what happened.

Start from lesson 1 Unlock every course

Curriculum

The 30 lessons are grouped into six chapters. Working through from chapter 1 is recommended, but you can also dip into the parts you need. * Standard-library Python runs right in the browser. All commands and logs come from fictional hosts.

Chapter 1 — What logs are for (lessons 1–5)

What logs are for and what makes them useful. You align timestamps and formats, and decide what must never be logged.

Why We Keep Logs

The four jobs of logs: detection, investigation, accountability and troubleshooting. Without a record, an investigator simply cannot tell what happened.

Free

What Makes a Good Log

Who, what, when, where and with what result. Timestamps with a time zone and a consistent format are what make a log useful.

Free

Keeping Clocks in Sync

Syncing clocks with NTP and chrony, storing in UTC and displaying in local time. Why clock skew breaks log correlation, and how to normalise timestamps.

Free

Log Formats: Plain Text, Syslog and JSON

Free-form text, RFC 5424 syslog and JSON structured logs. Their strengths and weaknesses, compared by actually parsing them.

What Not to Log

Keeping passwords, tokens and personal data out of logs. How to mask values, and how to choose log levels for production.

Chapter 2 — Linux logs (lessons 6–10)

Recording and reading what happens on Linux with journald, rsyslog and auditd. The chapter ends with retention and rotation.

journald and journalctl

Filtering the systemd journal with journalctl: by unit, priority and time, plus making the journal persist across reboots.

Configuring and Forwarding with rsyslog

Writing rules that route logs by facility and severity, and forwarding them to another server over TCP and TLS, with a queue to avoid gaps.

Reading Authentication Logs

How to read the sshd and sudo records in auth.log and secure. Counting failed logins per source IP and looking at them together with successes.

Recording Activity with auditd

Using the kernel audit system to record writes to key files and command execution. Writing rules, and reading the results with ausearch and aureport.

Log Rotation and Retention

Rotating logs with logrotate and capping journald. Deciding how long to keep logs, and estimating the disk space you need.

Chapter 3 — Application and network logs (lessons 11–15)

What web, DNS, proxy, firewall and database logs can tell you.

Web server access logs

The nine fields of the combined log format and how to read status codes. Parse an access log in Python and count the top paths and the sources with the most errors.

Error logs and application logs

Reading web server error logs and the security events an application should write. Use structured logs and request IDs to stitch several logs into one story.

DNS and proxy logs

What DNS query logs and proxy logs reveal, what suspicious lookups look like, and the privacy care needed when recording employees' traffic.

Firewall and flow logs

Logging allowed and denied traffic, the ideas behind NetFlow and IPFIX, and cloud flow logs. Aggregate flows in Python to find heavy senders and sources trying many ports.

Database and authorisation logs

Recording database logins, privilege changes and administrator queries. PostgreSQL settings and pgAudit, authorisation logs in the application, and balancing coverage against volume.

Chapter 4 — Windows and cloud logs (lessons 16–20)

Windows event logs and Sysmon, plus the audit logs of cloud platforms and identity providers.

Windows event log basics

Channels and Event Viewer, the first event IDs to learn (4624, 4625, 4688, 4720, 4732, 1102) and logon types, plus how to pull events out with PowerShell.

Audit policy and process creation logs

Turning on what is not recorded by default with advanced audit policy. Adding the command line to event 4688, sizing the log, and watching out for secrets in command lines.

Widening visibility with Sysmon

What Sysmon adds to the built-in logs (process, network, file and DNS events) and its main event IDs. How to approach its configuration, and using ProcessGuid to follow one process.

Cloud audit logs

What AWS CloudTrail, the Azure activity log and Google Cloud Audit Logs record, and how long they keep it by default. Setting up longer retention and watching for attempts to stop logging.

SaaS and identity provider logs

What sign-in logs, admin audit logs and MFA events tell you, how to cope with short default retention, and how to spot an MFA approval after a run of denials.

Chapter 5 — Collect and protect (lessons 21–25)

Bringing logs into one place, normalising them, and protecting them from tampering and gaps.

Bringing logs into one place

Why logs should not stay on each server, and a collection architecture of agents, forwarders and a central store, including encrypted transport and buffering.

Normalisation and common field names

Mapping each source's own field names onto a common schema. A mapping table to ECS-style names and a conversion in Python.

Protecting logs from tampering

Append-only storage, a separate write-only account, and detecting tampering with hash chains. Also journald sealing and cloud validation features.

Retention and cost

Splitting storage into hot, warm and cold tiers. Retention per log type, sampling and summarising, and estimating volume.

Monitoring the logging pipeline itself

Noticing logs that stop arriving: last-seen time per source, agent heartbeats and ingestion lag.

Chapter 6 — Monitoring and alerts (lessons 26–30)

Turning collected logs into day-to-day monitoring with dashboards and alerts. The course ends with a small monitoring plan.

What to put on a dashboard

A dashboard is a tool for noticing "different from usual". Knowing the normal shape, a few key panels, and avoiding vanity metrics.

Thresholds and baselines

Static thresholds versus judging by deviation from a baseline. Finding outliers with mean and standard deviation, and baselines per hour of day.

Preventing alert fatigue

How too many alerts bury the real ones. Severity and routing, deduplication, and writing alerts that lead to action.

On-call and runbooks

An on-call rota that decides who receives alerts, and runbooks that let them act without hesitation. Escalation steps, records and reviews.

Wrap-up: writing a small monitoring plan

Using a small fictional service, put logging, collection, retention, monitoring and on-call into one plan. A bridge to the Detection Engineering course.

Once you have finished all 30 lessons, move on to the Detection Engineering course. See the order of courses on the security track page. Membership unlocks every course.