Elastically (Re)laxin’ Cabanas: Local Observability
Who likes monitoring on a shoestring? I do! When it comes to cyber traffic monitoring, the ELK Stack stands as the industry standard to the best of my knowledge. It comprises a dynamically indexed database and analytics tool, Elastic. What it can do is quite impressive, as is the amount of resources it requires. Nevertheless, you can run a test-load-sized instance within a few Docker containers locally.
Using a single docker-compose.yml and some configuration files, you can allocate fewer resources for local configuration and then scale it up for deployment on Google Cloud Platform, other cloud providers, or your own server. Though, if you’re reading this, it’s unlikely you’re running your own hardware.
The “L” in ELK stands for Logstash, a data preprocessing stage. Its most powerful features when it comes to cyber monitoring include:
- Parsing logs from various sources and transforming them into a unified format.
- Filtering and enriching log data with additional context, such as GeoIP data for IP addresses.
- Aggregating logs from multiple sources and streaming them to Elasticsearch for storage and analysis.
Logstash receives “beats” which are either streamed continuously through a service like RedPanda, which we experimented with in the previous post, or in batches using “beats”. In our case, we’re using Filebeat, and our applications will generate logs, storing them in a location to be ingested.
“K” is for Kibana, a powerful dashboarding technology that performs functions like data visualization, search, and analysis. It is highly configurable and ideal for monitoring network traffic. Filtering by port, you can see increased traffic in unusual places. For example, anomalous traffic in egress from Port 53 could indicate the exfiltration technique used by adversaries called “DNS Tunneling.”
It is certainly convenient to experiment with these tools locally, and you can find the instructions here if you are interested in experimenting. Keep in mind that a production workload will consume resources and money like no tomorrow.
Adding these features to the project makes it feel more legitimate. I want to take a moment to acknowledge how impressive the Docker technology and ecosystem are. Once deployed, I look forward to understanding a little more about Kubernetes.
Peace and love,
Snyata
Aylex Riom 