Job Description

Job Description

Position: Databricks Data Engineer

Location: Remote (U.S. based) — Preference for candidates in or willing to relocate to Washington, DC or Indianapolis, IN for periodic on-site support

Citizenship Requirement: U.S. Citizen

Role Summary:

Seeking a Databricks Data Engineer to develop and support data pipelines and analytics environments within an Azure cloud-based data lake. This role translates business requirements into scalable data engineering solutions and supports ongoing ETL operations with a focus on data quality and management.

Key Responsibilities:

• Design, build, and optimize scalable data solutions using Databricks and Medallion Architecture.
• Develop ingestion routines for multi-terabyte datasets across multiple projects and Databricks workspaces.
• Integrate structured and unstructured data sources to enable high-quality business insights.
• Apply data analysis techniques to extract insights from large datasets.
• Implement data management strategies to ensure data integrity, availability, and accessibility.
• Identify and execute cost optimization strategies in data storage, processing, and analytics.
• Monitor and respond to user requests, addressing performance issues, cluster stability, Spark optimization, and configuration management.
• Collaborate with cross-functional teams to support AI-driven analytics and data science workflows.
• Integrate with Azure services including:
• Azure Functions
• Storage Services
• Data Factory
• Log Analytics
• User Management
• Provision and manage infrastructure using Infrastructure-as-Code (IaC).
• Apply best practices for data security , governance , and compliance , supporting federal regulations and public trust standards.
• Work closely with technical and non-technical teams to gather requirements and translate business needs into data solutions.

Preferred Experience:

• Hands-on experience with the above Azure services.
• Strong foundation in advanced AI technologies.
• Experience with Databricks , Spark , and Python.
• Familiarity with .NET is a plus.

Job Requirements:

To be considered for this position, candidates must possess:

Education & Experience:

• Bachelor’s degree in Computer Science or related field with 3+ years of experience, or
• Master’s degree with 2+ years of experience.

Technical Expertise:

• 3+ years of experience designing and developing ingestion flows for structured, streaming, and unstructured data using cloud platform services, with a focus on data quality.
• Databricks Data Engineer certification and 2+ years of experience maintaining the Databricks platform and developing in Apache Spark.
• Proficiency in Python , Spark , and R ; experience with .NET is a plus.
• Strong knowledge of data governance practices, including:
• Metadata management
• Enterprise data catalog
• Design standards
• Data quality governance
• Data security

Client Interaction & Communication:

• Ability to work directly with clients and provide front-line support.
• Skilled in documenting and presenting solutions using architecture and interface diagrams.

Development Methodologies:

• Experience with Agile methodologies.
• Familiarity with CI/CD automation and cloud-based development (Azure, AWS).

Preferred Qualifications (Not Required):

• Certifications in Azure Cloud.
• Knowledge of FinOps principles and cost management.

Eligibility:

• This role will require US Citizenship and the ability to obtain a public trust clearance. No C2C allowed.