When Your Git Repo Becomes a Database
Every security team has that Git repository.
It started innocently enough. A few configuration files, some threat intelligence, maybe a list of indicators or assets.
Then someone adds a script to collect data automatically. Another script consumes it. A pipeline starts depending on it. A security tool uses the repository as its source of truth.
At some point, you realize something slightly strange:
Your Git repository has become a database.
The commits are now your ingestion pipeline. And the pull requests are somehow part of your data management workflow.
Technically functional. Morally questionable.
Git is good at what Git does
Git is excellent at managing versions of files.
You can see what changed, when it changed, and who changed it. You can review changes before merging them and roll back to a previous state.
That makes Git useful for things such as:
- configuration
- detection rules
- automation code
- infrastructure definitions
- documentation
- other version-controlled artifacts
The problem starts when we use those properties to solve a different problem: storing and distributing frequently changing data.
Imagine a security pipeline collecting thousands of indicators every hour.
Instead of putting those indicators into a database, the pipeline writes them to a JSON file and commits the file.
Another system periodically pulls the repository and reads the JSON.
Now Git is storing the data, distributing the data, and acting as the communication mechanism between systems.
It works.
But the architecture is now built around the wrong abstraction.
The problem is not Git
Git is not inherently bad for storing data.
There are legitimate cases where security data belongs in a repository.
For example, a set of detection rules may benefit from version history and code review. A configuration file may need exactly the same properties.
The problem is using Git as a substitute for infrastructure that already exists for another purpose.
Consider a simple data flow:
The repository is now effectively acting as the interface between the collector and consumer.
Git was designed around versioning files, not continuously exchanging application data.
That difference becomes important as the system grows.
Choose the pattern based on the problem
You are storing data
Use a database.
If the system needs to query, update, filter, or correlate structured data, a database is a better abstraction.
Depending on the requirements, that might be a relational database such as PostgreSQL, or a NoSQL database.
The point is to separate data storage from source-code version control.
Instead of:
you can have:
The data can be queried directly without treating a commit as a database operation.
You are sharing data
Use an API.
If several systems need access to the same data, an API gives them a defined interface.
For example:
Consumers do not need to know how the underlying data is stored.
They only need to understand the API contract.
An API also gives you a natural place to handle authentication, authorization, validation, filtering, and pagination when needed.
You are pushing updates
Use messaging.
If the requirement is:
"Tell another system that something changed."
then a queue or pub/sub system is often a better fit than periodically checking a Git repository.
For example:
The event itself is the unit being communicated.
Consumers do not need to repeatedly ask whether a repository changed.
You are deploying something
Use CI/CD.
Sometimes a repository is being used as a database because the actual requirement is to distribute a new version of something.
In that case, Git and CI/CD may be exactly the right tools.
For example:
Here Git is doing what it is good at: versioning the desired state.
The pipeline is responsible for turning that state into a deployed artifact or configuration.
That is different from committing every newly collected piece of runtime data into the repository.
The difficult part is knowing which problem you have
Security automation often starts with a small script.
You need to get data from A to B, so you write something simple:
There is nothing inherently wrong with starting this way. The problem is when the workaround becomes infrastructure. Soon you need another consumer. Then you need to know when the data changed. Then someone needs historical data. Then another process starts polling the repository. Then you need to deal with merge conflicts, repository size, polling intervals, stale checkouts, or delayed updates.
None of these necessarily make Git unusable.
They are signals that the original problem may no longer be versioning files.
It may now be moving and storing application data.
Think like a developer
Security engineers do not need to become software engineers.
But when we build automation, we should borrow some of the patterns that software engineers use to design systems.
The question should not simply be:
"Can I automate this?"
It should also be:
"What kind of system am I actually building?"
If you are storing data, think about data storage. If you are exposing data to other systems, think about APIs. If you are propagating events, think about messaging. If you are deploying changes, think about CI/CD.
And if Git is involved, ask whether version control is actually the requirement, or simply the easiest thing available.
That distinction can prevent a small automation script from quietly becoming a very strange distributed system.
Conclusion
There is nothing wrong with using Git in security automation.
There is something questionable about making Git impersonate a database, message broker, API, and deployment system at the same time.
The practical lesson is simple:
Use the tool that matches the problem.
Security engineers already build systems that collect, process, and distribute large amounts of data.
We do not need to reinvent software engineering patterns every time we automate something.
Sometimes the best improvement to a security automation is not a more complicated script.
It is recognizing that the script has become a system — and designing it like one.