Job Title: Technical Lead – Data Engineering (Palantir, Spark, PySpark, Python)

Please dont apply if you dont have PALANTIR experience

Onsite work :One or two days in a month W2 role OPT with 7 years exp is fine

Job Summary

We are looking for a hands-on Technical Lead in Data Engineering to drive the design, development, and delivery of scalable data solutions in a large retail enterprise. This role requires strong expertise in Palantir Foundry, Spark/PySpark, SQL, and Python, along with the ability to lead engineering teams and partner with business stakeholders across supply chain, merchandising, and store operations.

Key Responsibilities

Lead the design and implementation of scalable data platforms and pipelines using PySpark, Spark, and Python

Drive adoption and best practices for Palantir Foundry (data pipelines, ontology, workflows, and operational applications)

Architect and optimize high-performance data processing solutions for large-scale datasets

Provide technical leadership and mentorship to data engineers, ensuring code quality and best practices

Collaborate with cross-functional teams (business, analytics, data science) to translate requirements into scalable solutions

Design robust data models, ETL/ELT frameworks, and data integration strategies

Ensure data quality, governance, security, and compliance across enterprise data platforms

Lead performance tuning, troubleshooting, and optimization of data pipelines

Drive CI/CD implementation, code reviews, and release management

Stay current with emerging data engineering technologies and recommend improvements

Required Qualifications

Bachelor’s or Master’s degree in Computer Science, Engineering, or related field

8+ years of experience in data engineering, with at least 2–3 years in a technical leadership role

Strong hands-on experience with:

Python & PySpark

Apache Spark

Advanced SQL

Experience with Palantir Foundry or similar modern data platforms

Deep understanding of data engineering principles (ETL/ELT, data modeling, distributed systems)

Experience designing and managing large-scale data architectures

Strong leadership, communication, and stakeholder management skills

Preferred Qualifications

Experience in retail domain (supply chain, inventory, merchandising, store analytics)

Experience with cloud platforms (Azure, AWS, or GCP)

Familiarity with orchestration tools (Airflow, Azure Data Factory, etc.)

Experience with real-time/streaming data pipelines

Exposure to DevSecOps practices

Key Skills

Technical Leadership

Data Engineering Architecture

PySpark & Spark

Python Programming

SQL Optimization

Palantir Foundry

Data Modeling & Warehousing

CI/CD & DevOps

Flexible work from home options available.