The Log Must Flow
Building a security data pipeline with Kafka and Apache Flink
My LinkedIn cover photo says "the log must flow". It's a reference to Dune, where spice must flow because it powers the Empire's economy: transportation, computation, everything runs on it. Stop the flow and the system stalls.
Security logs work the same way for security operations. They power detection, investigation, and response. Stop the flow and those stop too.
That's where the title comes from. What this post is actually about is what "flow" should mean for logs in practice.
Logs are generated continuously, consumed by multiple systems, enriched with additional context, transformed into different schemas (or the OCSF schema — see Why OCSF is awesome), and eventually used to make security decisions.
Yet many security architectures still treat logs primarily as something to store and query.
A log arrives.
It gets indexed.
Later, someone needs additional context, so they query the stored data, export the results, enrich them, and sometimes write the result somewhere else.
That works.
Until the amount of data makes repeated processing expensive, slow, or unnecessarily complicated.
My argument is simple:
If a transformation can be performed while the event is flowing through the system, it is often better to do it there than repeatedly after the event has already been stored.
Kafka gives us the event stream.
Flink gives us the processing layer.
The downstream analytics system receives the resulting security data.
What happens between those stages matters.
Over 100,000 organizations run Kafka, including 80%+ of the Fortune 100. Alibaba processes around 40 billion events/day through Flink; Netflix, Uber, and ING run it too.
The problem with processing everything after ingestion
Consider a fairly ordinary network event:
{
"timestamp": "2026-09-06T10:15:42Z",
"src_ip": "203.0.113.42",
"dst_ip": "10.10.20.15",
"dst_port": 443,
"action": "allowed"
}
There is useful information here, but there is also information missing.
For example:
- Which ASN is associated with the source IP?
- Which organization owns the address?
- Which country is it associated with?
- Is it known to threat intelligence?
- Is the destination asset externally exposed?
- Who owns the destination asset?
Some of this information can be added before the event reaches the downstream analytics system. Some can be added later. The architectural question is where and when that enrichment should happen.
One possible architecture is:
The problem becomes more obvious when the same enrichment is required by multiple searches. Suppose the same IP appears in thousands of events. If every investigation or scheduled detection independently performs the lookup, the same piece of contextual information gets calculated repeatedly. There is no fundamental reason for that when the enrichment is stable enough to attach to the event earlier in the pipeline.
This is not unique to SIEMs. Modern analytical systems also provide mechanisms for transforming and enriching data closer to ingestion — Elasticsearch's ingest pipelines and enrich processors, or ClickHouse's materialized views, are two examples.
From a storage system to a data flow
Instead of making the SIEM the center of every operation, we can introduce an intermediate event stream:
Now the event has a lifecycle. It is generated once. Kafka provides the event stream. Flink processes the stream. The downstream system receives the result. The downstream analytics system is no longer required to be the place where every transformation happens.
Why Kafka and Flink specifically
Two properties matter here, and they're the reason this argument leans on Kafka and Flink rather than "a queue" and "a processor" in the abstract.
Kafka separates producing an event from processing it, and it keeps events around. A topic is partitioned, so consumers can process partitions in parallel. The processing layer can scale independently of whatever is generating the events, subject to partition count and processing capacity. Kafka also retains events for a configurable period rather than deleting them on consumption. A processing application can therefore restart without losing its input, and old events can be replayed if the processing logic changes.
Flink adds state on top of that stream. Some transformations only need the current event: parse a timestamp, normalize a field. Others need memory. Counting authentication failures for an account within a time window, for example, requires the processor to remember previous events. Flink's keyed state provides that memory, while checkpointing allows state and stream position to recover after a failure.
That combination — replayable input plus recoverable state — makes it reasonable to put stateful logic in the stream instead of doing it at query time.
Processing can happen while the event is already moving through the system:
A security event in motion
Suppose a firewall generates:
{
"timestamp": "2026-09-06T10:15:42Z",
"src_ip": "203.0.113.42",
"dst_ip": "10.10.20.15",
"dst_port": 443,
"action": "allowed"
}
It enters Kafka, and Flink consumes it and performs transformations that are useful to downstream consumers: normalize, enrich, correlate.
The resulting event might look like:
{
"timestamp": "2026-09-06T10:15:42Z",
"src_ip": "203.0.113.42",
"src_asn": 64500,
"src_country": "NL",
"dst_ip": "10.10.20.15",
"dst_asset": "web-01",
"dst_owner": "infra",
"dst_port": 443,
"action": "allowed"
}
The downstream system now receives an event that already contains the context required by multiple consumers. Enrichment happens at a defined processing stage instead of being independently repeated by every consumer that needs the same context.
Enrichment is where the architecture becomes interesting
The repeated-lookup problem above gets more concrete once you look at what a single enrichment actually costs at scale. Suppose the stream contains the same source IP over and over:
203.0.113.42
203.0.113.42
203.0.113.42
...
A naive implementation would perform an external lookup for every event. There is little value in that when the underlying data — ASN, organization, country — is stable enough to reuse.
The lookup result needs to be stored somewhere so subsequent events can reuse it: Flink state, an external cache, or another reference-data mechanism, depending on requirements.
The exact implementation depends on the enrichment source and its consistency requirements. For relatively static reference data, caching prevents repeated external requests. For data that changes frequently, the design gets more complicated because a cached enriched event can go stale. Not every enrichment belongs in a streaming pipeline — Elasticsearch's enrich processor, for example, is intended for reference data that doesn't change frequently, for the same reason.
But what about the raw event?
What happens if your enrichment logic is wrong? What if the enrichment source is unavailable? What if tomorrow you decide that a field you discarded was actually important?
This is why you shouldn't treat the pipeline as:
A better model is:
The raw stream provides a recovery mechanism. If the processing logic changes, retained events can potentially be replayed through a new version of the processing pipeline, depending on the retention and replay strategy.
Kafka's model supports reading retained events again rather than treating them as immediately deleted after consumption. Flink's fault-tolerance model also relies on replayable input streams together with checkpoints when recovering stateful applications.
Retention must be part of the design. If replay matters to you, your system needs enough retained data to make that replay possible.
Backpressure: streaming does not remove bottlenecks
Moving processing into a stream does not make expensive operations disappear.
Imagine Kafka pushing 100,000 events/sec into Flink, which then has to call an external enrichment service that can only handle 5,000 requests/sec. That enrichment service is now your bottleneck — the streaming architecture has just made it explicit instead of hiding it behind a dozen independent queries.
Your processing layer can apply mechanisms such as caching, batching, asynchronous I/O, rate limiting, and controlled concurrency instead of letting every downstream consumer independently overload the same dependency. Those mechanisms have their own trade-offs. Caching introduces freshness considerations. Batching introduces latency. Asynchronous processing introduces concurrency and failure-handling concerns. Rate limiting reduces pressure on the dependency but can increase queue depth.
Streaming moves processing into a place where throughput, state, and failure behaviour can be explicitly managed. It doesn't make the underlying cost go away.
This doesn't mean everything should be streaming
There is a temptation with architectures like this to turn every problem into a Kafka topic and every transformation into a Flink job. That would be a mistake.
Streaming introduces its own operational complexity: partitioning decisions, state management, schema evolution, backpressure, checkpointing, failure recovery, external dependency failures, monitoring, deployment and upgrades. Reference data also needs careful treatment — an enrichment that changes every few seconds is a poor fit for a mechanism built around relatively static reference data, for the same reason discussed above.
The question you should ask before adding a streaming component is: does processing this information as it flows provide a meaningful advantage over processing it later? If the answer is no, the component only adds complexity.
Let the log flow
A SIEM is an important part of a security architecture. But it does not have to be the place where every security-data operation happens.
When event processing is separated from event storage, Kafka can provide the durable event stream and Flink can provide stateful processing. The downstream analytics system can then consume the processed events for investigation, hunting, detection, and analysis — where the log does not have to wait until someone queries it before becoming useful.