Job Description

We are seeking a skilled Azure Databricks Engineer to design, develop, and optimize large-scale data processing systems using Azure Databricks. The ideal candidate will have expertise in Apache Spark , data engineering pipelines, and Azure cloud technologies. You will collaborate with cross-functional teams to build robust, secure, and scalable data solutions.

Typical Duties and Responsibilities:

• Design and implement data ingestion and transformation pipelines using Azure Databricks and other Azure data services.
• Develop ETL/ELT processes for structured, semi-structured, and unstructured data.
• Optimize and tune Apache Spark jobs for performance and cost efficiency.
• Build and manage scalable data lakehouse solutions using Delta Lake and Azure Data Lake Storage.
• Integrate Databricks with Azure Synapse Analytics, Data Factory, and other Azure resources.
• Implement security best practices: role-based access control, encryption, and data masking.
• Collaborate with data scientists and analysts to operationalize machine learning models using MLflow.
• Automate workflows with Databricks Jobs and CI/CD pipelines.
• Monitor and troubleshoot performance issues in Databricks clusters and Spark applications.

Qualifications

We require 2+ years of experience in the following areas:

• Hands-on experience with Azure Databricks and Apache Spark for large-scale data processing.
• Strong programming skills in Python, Scala, and SQL.
• Expertise in Azure Data Lake, Blob Storage, Azure Synapse, and Azure Data Factory.
• Hands-on experience with CDC tools and frameworks, including but not limited to Debezium, SQL Server CDC, or similar technologies.
• Expertise in configuring and managing CDC pipelines within Azure cloud.
• Experience with Delta Lake architecture for data reliability and performance.
• Knowledge of Spark job performance tuning and optimization strategies.
• Strong understanding of data security and governance in cloud environments.
• Experience with CI/CD for Databricks and Infrastructure as Code (Terraform, ARM templates).
• Excellent problem-solving skills and ability to work in a collaborative team environment.

Education and Certification:

• Bachelor’s degree in IT or related field is highly preferred.
• Databricks Certified Associate Developer for Apache Spark or related certification is preferred.
• Microsoft Azure Data Engineer Associate Certification is preferred.

Additional Information

What you should know about us:

• We are humble, hungry, and smart. We solve big problems, serve lots of clients, and are entirely committed to delivering transformative outcomes.
• We are team players, deeply dedicated to the mission of the organization, and to helping everyone around us be successful.
• We compensate well, rewarding performance that delivers positive outcomes for our clients and ensuring incentives are aligned to achieve our goals.
• Our leaders work hard, serving as shining examples of what it means to live out our values. They are servant leaders, helping their teams to be successful in all possible ways.
• We have a great benefits package including unlimited vacation, significant 401k contributions, and several opportunities to develop yourself.
• We pride ourselves in having the best talent in the industry and hope that you're up for the challenge!

What our team members say about us:

• "I love our true empathy and concern for our clients, it's very rare and appreciated. It is a pleasure to be a part of an organization like this."
• "I learn something new every single day, and I feel like I'm a part of building an organization that has legs. I appreciate that I'm consistently humbled by the talent and caliber of our team."
• "The culture of the company is amazing, and the climate of my team is great. The benefits that employees are offered are better than competitors, and the one-on-one presence that my team lead gives is extremely beneficial to me."

All qualified applicants will receive consideration for employment without regard to age, color, sex, disability, national origin, race, religion, or veteran status.

Equal Opportunity Employer