Location: Onsite – Houston Employment Type: Full-Time

About the Role

We are seeking a highly skilled, hands-on Databricks Lead with 8+ years of experience in data engineering, including deep expertise in Azure Databricks, PySpark, and Structured Streaming. This role is ideal for a senior engineer with a programming-first mindset, a strong understanding of distributed systems, and the ability to build high-performance, cost-optimized data solutions.

This is not a traditional ETL role. It requires extensive knowledge of Spark internals, declarative pipeline development, and real-time data processing. The successful candidate will also provide technical leadership and collaborate closely with cross-functional teams to deliver robust, scalable data solutions.

Key Responsibilities

Lead the design and development of scalable, high-performance data pipelines, streaming tables, and Delta Live Tables (DLT) using Azure Databricks and PySpark

Drive Spark performance tuning and implement cost-optimization strategies within the Databricks environment

Build and manage real-time and batch workflows using Structured Streaming

Leverage Delta Live Tables (DLT) and Lakehouse Declarative Pipelines (LDP) to build scalable, reliable, and maintainable data pipelines

Collaborate with data architects, analysts, and business stakeholders to deliver robust data solutions

Provide technical leadership through mentorship, code reviews, and architectural guidance

Ensure system reliability and performance through proactive monitoring and engineering best practices

Must-Have Qualifications

8+ years of hands-on experience in data engineering, big data development, or a related field

Extensive hands-on experience with Azure Databricks and PySpark

Strong programming background and in-depth knowledge of distributed data processing beyond traditional ETL tools

Proven expertise in:

Spark performance tuning and optimization

Cost management within Databricks

Structured Streaming for real-time data processing

Lakehouse Declarative Pipelines (LDP) and Delta Live Tables (DLT)

Familiarity with the Azure data ecosystem, including ADLS, Azure Data Factory, and Synapse

Excellent communication skills and the ability to collaborate effectively with cross-functional teams

Willingness and ability to work onsite in Houston

Nice-to-Have Qualifications

Industry experience in energy, utilities, or heavy industry

Knowledge of data governance, security, and monitoring