- Configured automated data ingestion processes via ADF, which decreased manual data entry hours by 30 hours monthly
- Created numerous pipelines in Azure using Azure Data Factory V2, utilizing activities like Move & Transform, Copy
- ForEach, Filter, and Databricks, reducing pipeline execution time by 60
- Worked on Databricks using PySpark and Spark-SQL, improving data transformation speed by 70% for data cleaning
- Exposed transformed data in the Azure Spark Databricks platform to Apache Parquet and Delta file formats for efficient data
- Configured and implemented Azure Data Factory triggers and scheduled pipelines, resulting in a 40% reduction in data
- Designed and developed Azure Logic Apps to trigger emails in case of pipeline failures in Azure Data Factory, reducing the pipeline failure rate by 70
- Enhanced data validation processes by 60%, ensuring data quality across the Bronze, Silver, and Gold Zones in Azure Data
- Lake Gen1/Gen2
- Created interactive data visualizations in Power BI that transformed complex data sets into clear graphical representations
T
Tiger Analytics
Data Engineer Intern
Chennai, IN-TN, IN
August 2021 - January 2022
Skills
Languages
EnglishTamil
Skills
Access ControlsAlgorithmsAnalytical ThinkingApache HadoopApache HTTP ServerApache SparkAutomationAzure Data FactoryAzure Data LakeBig DataBusiness RequirementsDashboardsData AnalysisDatabasesDatabricksData CleansingData IngestionData PipelinesData ProcessingData QualityData SecurityData StreamingData TransformationData ValidationData VisualizationDecision Making SkillsDecision TreesExtract Transform Load (ETL)Forecasting SkillsHard Work and DedicationInformation EngineeringInformation TechnologyJava (Programming Language)Knowledge of EngineeringLinear RegressionLogistic RegressionMachine LearningManual Data EntryMatplotlibMetricsMicrosoft AzureMicrosoft SQL ServerMySQLPower BIProgramming LanguagesPysparkPython (Programming Language)Random ForestRole-Based Access ControlScikit LearnSoftware LibrarySQL AzureSQL DatabasesSql Data WarehouseSQL Server Integration ServicesStakeholder ManagementSupport Vector MachineUser ExperienceVacuum Cleaners