Ingesting, Processing, and Distributing “Intelligence”🦀🔥
Everything that we plan to do with the Orion platform, and lets face it any near real-time Machine Learning Project require high-speed, multi-source, and reliable pipelines to ingest, process, and distribute data. Whether it’s application or system logs, packets in an internal network, or threat intelligence API feeds, the ability to process and analyze this data in real-time, find sneaky anomalous data and provide accurate triage and direction is essential. This plan-inabox-bloggery outlines how we plan to build a high-capacity data pipeline in Rust (cough and Python), with an efficient ingestion layer, real-time processing, and distributed, specialised intelligence reports.
It exists but while we are still refining the platform why not get a bit rusty? Learning to manage concurrency without impacting the infamous runtime performance and memory safety of Rust is a great opportunity for THE FUN.
We’ll add a caching layer with Redis, leverage Docker Swarm for managing scale (Hi, K8s), and use Cloudflare Workers with Redpanda for globally distributed endpoints and streamed message processing. Ultimately, these shenanigans result in a mostly serverless pipeline leveraging Cloudflare’s enormous footprint and Google Cloud Platform’s POWAAA.
Features
The Plan so far:
- Multi-source data ingestion: Collect logs, PCAPs, and API data from global nodes in real time.
- Data processing and normalization: Using a multi-threaded Rust system for high-speed parallel data processing.
- Cloudflare Workers and Databases: Cloudflare’s globally distributed network of endpoints will form the nodes in the dragnet (theoretically of course – though the workers are perfect candidates and the regionalized AI Workers are ideal candidates.
- Redpanda on GCP: For globally distributed message queuing.
- Redis caching: Caching intermediate data for improved performance.
- Intelligence reporting: Generating department-specific reports from processed data.
- Scalable architecture: Using Docker Swarm (and/or) Kubernetes to manage the pipeline at scale. We’ve never handled big enough workloads for the use of these.
Managing Scale with Docker Swarm w/ Eyes on Kubernetes
Our data pipeline needs to handle heavy loads across multiple data sources and scale dynamically. While Kubernetes is a go-to for many scalable architectures, we’ll use Docker Swarm for simplicity, with a view to transition to Kubernetes as the system grows.
Plans Are Useless, Planning is Compulsory:
Scale 1: Setting up REDIS and Docker Swarm
This caching layer will ensure frequent queries and intermediate results are quickly accessible without needing reprocessing.
Scale 2: Scaling with Docker
In a Docker Swarm environment, the Rust processing service can be scaled horizontally:
Each service (Redpanda, Redis, and our Rust processors) runs in containers that can scale based on load. We’ll deploy Redis as a service to handle cache invalidation and data locality, ensuring fast access to frequently queried or processed data.
Google Cloud Platform + Cloudflare Edge Workers: A Love Story
To handle the globally distributed data and ensure optimal performance, we’ll deploy Cloudflare Workers for edge computing and couple that with Google Cloud Platform (GCP) for centralized data processing.
GCP Centralized Processing
In GCP, Redpanda acts as the primary message queue, sending data to our Rust-based backend for heavy processing. GCP Compute Instances handle the raw processing load, and Cloud Run can serve specific REST APIs out of containers for more serverless fun.
This setup allows edge ingestion with global coverage through Cloudflare Workers, while GCP handles the processing and Redpanda distributes messages efficiently.
Detecting & Reporting on Worldwide Nodes
Once data is ingested and processed, intelligence reports can be generated for various departments (e.g., security, operations, legal). These reports are compiled from normalized data, using department-specific filters and enrichment.
Core reporting to the infrastructure will remain instantaneous but these reports will be merged with existing threat intelligence to enrich the data and provide deeper insights.
Intelligence Report Generation
In our Rust backend, we’ll produce mundane reports that adhere to the ECS schema forom for use in common SIEM’s and ELK based reporting.
We’ll then use our agentic-analysis system to focus reports based on department needs. For example, the security team might get a report focused on network threats, while the legal team gets logs of access violations.
Step 2: Distributing Reports
Using Redis for quick access to a Binary Hash for O(1) lookup comparisons and a Kafka producer (or Redpanda) for sending out these department-specific reports.
This ensures each department receives tailored, timely reports through their dedicated channels.
Wish Us Luck
Building a high-capacity data pipeline with Rust, Redpanda, and Cloudflare Workers is core to our system. It combines the best of modern technologies: Rust’s performance, Cloudflare’s global edge network, and Redpanda’s Kafka compatibility. By leveraging Docker Swarm or Kubernetes, we ensure scalability and reliability across distributed environments. Redis provides the necessary caching for speed, while our report generation and distribution engine ensures that actionable intelligence reaches the right hands.
In this journey, we’ve optimized for performance and global scale. Now, all we need is a little bit of luck as we deploy this robust system into production. Data lineage and governance are back of mind as the use of data from a region protected by for example GDPR requires it remain in that region.
That’s a good problem to have if we come to that and we’ll wait for that day.
— Snyata
Aylex Riom 