Audit logging architecture.
An audit log that can be edited is not an audit log, which makes this one of the few systems where the absence of features is the design.
The components
Every row below is read from the graph that produced the diagram above, so the two cannot disagree.
| Component | Tier | Why it is there |
|---|---|---|
| Application services | Client | Supporting component |
| Audit SDK | Application | One schema, so events are comparable across services |
| Ingest API | Application | Append only, no update or delete path exists |
| Hash chain | Application | Each record signs the previous, so gaps are detectable |
| Redaction | Application | Strips secrets before anything is written |
| Query API | Application | Read path is separate and heavily restricted |
| Retention worker | Application | Enforces the policy; deletion is itself audited |
| Kafka | Data | Supporting component |
| Append-only store | Data | Supporting component |
| Object storage | Data | Write-once with object lock |
| SIEM | External | Supporting component |
Design decisions worth arguing about
A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.
Append-only means no update path exists at all
Not a convention, not a permission: there is no code that modifies a record and no endpoint that could. If a correction is needed it is a new record referring to the old one. That makes some perfectly reasonable operations impossible, including fixing a typo in a stored field, and that is the point. A log with an edit path has to prove it was not used; a log without one does not.
Hash chaining makes deletion detectable
Each record includes a hash of its predecessor, so removing or altering one breaks the chain from that point forward. It does not prevent tampering, since anyone who can write can recompute, but it makes silent tampering impossible for anyone without full write access, and it converts a deleted record into a visible gap. The cost is that records must be written in order, which constrains parallel ingest.
Redact before writing, never after
A secret in an append-only store is permanent by construction, which is exactly the wrong combination. Redaction has to happen on the ingest path before anything is durable. The consequence is that redaction rules are load-bearing security code with no second chance: missing one is not a bug you can fix later.
Separate the read path and keep it small
Audit logs are written constantly and read rarely, usually during an investigation or a compliance review. Keeping the query API separate and heavily restricted means the surface that could expose sensitive history is small and independently auditable. It also lets the write path stay fast, since it never has to serve a query.
Retention is a deletion policy, and deletion is audited
Regulations require keeping records for a period and privacy law often requires not keeping them beyond it, so a retention worker is mandatory rather than optional. The subtlety is that its deletions must themselves be audited, or the retention job becomes an unlogged way to remove history.
How it changes with scale
Volume grows with user actions rather than with traffic, so it is lower than application logging and retained far longer, often for years. Storage is the dominant cost and tiering matters: recent records queryable, older records in write-once object storage with a lock, and nothing deleted before the policy says so.
Where it breaks first
The log becoming unqueryable long before it becomes full. Years of records with no useful index turn every investigation into a full scan, and the log fails at the one moment it exists for. Deciding which fields are queryable at design time is the decision that determines whether this system is useful in three years.
Draw this yourself
Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.
When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.
Questions about this design
How is an audit log different from application logging?
Application logs are for debugging, are high volume, and are deleted quickly. Audit logs record who did what to which resource, are lower volume, retained for years, and must be tamper-evident. Mixing them means either paying audit costs on debug volume or losing the guarantees.
Do you need cryptographic signing?
Hash chaining is enough to detect tampering after the fact and is cheap. Full signing with an external timestamping authority matters where the log has to be evidence for someone who does not trust you.
What belongs in an audit record?
Actor, action, resource, timestamp, outcome, and enough request context to reconstruct what happened. Not the payload, which is where secrets end up.
Can audit records be deleted for a data subject request?
That is a genuine conflict between privacy law and retention obligations, and it is a legal question rather than a technical one. Architecturally, keep personal data referenced by an id you can delete separately, so the audit record survives while the identifying data does not.
More templates
- Vector Search Service DesignThe failure mode here is not an error. It is recall dropping slowly as the index grows, with nothing in your metrics saying so.
- AI Agent Orchestration DesignThe orchestrator owns the loop, and almost every hard problem here is about knowing when to stop.
- URL Shortener System DesignThe canonical read-heavy system: roughly a hundred reads for every write, and a redirect that has to be fast enough that nobody notices it happened.
- Chat Application System DesignLong-lived connections change everything: the hard part is not storing messages, it is knowing which of your servers is holding the socket you need to write to.
Last updated