Senior Data Engineer

MUST be local to Iselin NJ, Hybrid Schedule at least 3 days onsite

Salary: up to $160k 

No Sponsorship Available

Major Responsibilities:

• Building and maintaining data pipelines to ingest data from source systems into Databricks
• Designing, Building and maintaining generic frameworks to ingest data through the medallion layers
• Working with the core data platform team to integrate the framework with reference architecture
• Assisting the data mesh teams with best practices and onboarding to the new framework

Requirements:

• 10+ years of data engineering experience in building OLTP and OLAP databases, both on-prem and in the cloud
• 5+ years of experience with large scale data processing using SQL, Python, Pandas and PySpark
• 5+ years of experience building robust and scalable data pipelines in Databricks Lakehouse platform on any key cloud platform provider, preferably Azure cloud
• Experience with implementing Medallion architecture for data processing
• Experience with implementing Streaming data ingestion using Kafka
• Test driven development experience
• Proven experience in solution design and applying best design practices
• Data modelling and ER/UML diagram skills
• Experience with orchestration/enterprise scheduler platforms like Airflow, Tidal etc.
• Experience with implementing API based data integration pipelines
• Experience with source control and DevOps for CI/CD
• Experience working in Agile/scrum methodologies
• Experience with Jira, confluence for project work and documentation
• Experience with Service now for incident management
• Finance domain knowledge in Investment Banking (Securities, Capital markets)and Security reference data knowledge
• Nice to have ADF and SSIS experience

* College degree required.