- Automotive Product Catalogue Generation
- Incorporated Spark SQL in python for extraction, data preprocessing and stored parquet files in Azure Delta lake
- Utilized Databricks 'Job Runs' scheduler for leveraging one pipeline for multiple catalogue generation slashing
- OBD and VeCAN data processing
- Orchestrated processing and transformation of unstructured vehicle IoT sensor data from Azure IoT Hub, enhancing
- Logistic Operating System (L.OS
- Engineered authentication and authorization mechanism stage of data pipeline to ensure secure access control and data
- Created version control system as a common repository for Lambda functions employing AWS S3 and Lambda Layers
- Devised robust error handling and retry mechanism for data retrieval and manipulation using AWS SNS and SQS