Next-Edge Smart Security Cameras
High-Volume Event Streaming & Observability with AWS, Kafka & Kubernetes
Designed and implemented streaming infrastructure for high-volume event processing from IoT camera systems using Apache Kafka, AWS MSK, OpenSearch, and Kubernetes.
The Challenge & Context
High-volume event streams from thousands of deployed smart security cameras were causing ingestion bottlenecks, dropped messages, and high search latency during security incident investigations.
Engineering Approach
Designed and deployed a resilient event-driven architecture using Apache Kafka and AWS MSK to buffer incoming telemetry, paired with an OpenSearch/ELK cluster deployed on Kubernetes for sub-second log querying and NLP search integration.
Components & Data Flow
Camera IoT gateways push event payloads to AWS MSK (Managed Streaming for Kafka). Logstash workers running on Kubernetes consume Kafka topics, enrich event metadata, and index documents into Amazon OpenSearch. A Python FastAPI service exposes secure endpoints for frontend event querying.
Key Trade-offs & Decisions
AWS MSK over Self-Managed Kafka on EC2
Managed Kafka eliminates the operational burden of ZooKeeper/KRaft cluster maintenance, automated broker patching, and storage volume rebalancing.
Kubernetes-Hosted Logstash over Serverless Ingestion
Containerized Logstash on Kubernetes provides granular horizontal pod autoscaling based on Kafka consumer group lag, keeping compute costs 60% lower than serverless ingestion.
Obstacles & How They Were Overcome
RESOLUTION:Implemented custom partitioning keys based on camera zone hashes to ensure even workload distribution across all MSK broker instances.