Serverless Data Pipeline

🎯 Project Overview

Built a fully serverless data processing pipeline handling real-time IoT telemetry from 50,000+ devices. Processes 10 million events daily with automatic scaling and cost optimization.

🏗️ Architecture Highlights

• Azure Event Hubs for high-throughput event ingestion
• Azure Functions with consumption plan for cost-effective processing
• Azure Data Lake Storage Gen2 for raw and processed data tiers
• Azure Stream Analytics for real-time aggregations
• Power BI embedded for real-time dashboards

📊 Results & Metrics

• Processing latency under 100ms end-to-end
• 99.95% data reliability with dead-letter queue handling
• Cost reduced by 60% compared to VM-based solution
• Automatic scaling from 0 to 1000 concurrent executions