Sorry, this listing is no longer accepting applications. Don’t worry, we have more awesome opportunities and internships for you.

Lead Cloud Data Engineer (Data as a Service Enterprise Platform)

CardinalHire

Lead Cloud Data Engineer (Data as a Service Enterprise Platform)

San Francisco, CA
Paid
  • Responsibilities

    NO RECENT GRADS OR CONTRACTORS

    LOCAL CANDIDATES ARE PREFERRED!

    Our client is looking to hire experienced LEAD (BIG DATA) CLOUD ENGINEER to play a key role in designing and developing batch and real time data acquisition pipelines and analytics platform using latest open source big data technologies.

    In this role candidate will report into Data Engineering Director in the west coast (San Francisco) office. DATA AS A SERVICE (DAAS) is an enterprise platform team that organizes the client’s data and makes it available for applications and business units.

    It collects, correlates, enriches, and manages core data to produce single view of truth for use at speed and at scale. This group of passionate technologists thrive on simple and elegant architecture and agility.

    The client own the architecture and development of comprehensive data management, bringing real-time operational use, near real-time visibility use, and off-line deep analytical use together in a cohesive and holistic way. Innovation and engineering prowess is our core to stay ahead of our user’s demand and the industry changing landscape.

    KEY RESPONSIBILITIES:

    • Design and develop scalable data pipelines for optimal ingestion, transformation, storage and computation using latest big data technologies.
    • Work with engineering, product, and business team(s) to understand requirements, evaluate new features and architecture to help drive decisions.
    • Design, develop and maintain automated ETL processes for efficient batch records-matching of multiple large-scale datasets.
    • Translate complex business requirements into a scalable, efficient and high availability data platform. Strong ownership of the full backend stack - from design through to deploy and beyond.
    • Experience working with CONTINUOUS INTEGRATION (CI/CD) framework, building regression-able code within data world using GitHub, Jenkins, Cloud Build, Spinnaker and related applications.
    • Actively participate in code review and test solutions to ensure it meets best practice specifications. Prototype ideas quickly using cutting edge and new generation technologies.
    • Build collaborative partnerships with architects, technical leads and key individuals within other cross functional groups. Work with business teams/engineers to define instrumentation and data requirements.
    • Lead projects independently and also mentor team members and other engineers in the team.

    SKILLS SUMMARY:

    • Bachelor’s or Master’s degree or equivalent in computer science or related field with minimum of 6+ years of directly related work experience.
    • Good understanding of data structures and algorithms.
    • Programming experience in PYTHON, JAVA OR SCALA.
    • Extensive experience with ELASTIC STACK, KAFKA AND SPARK
    • Experience with big data technologies like HADOOP, MAPREDUCE, HIVE, SPARK/STORM, BIG QUERY
    • Experience with time series and NOSQL DATABASES IS A PLUS.
    • Experience working with CONTAINERS AND KUBERNETES is good to have.
    • Experience designing, implementing and scaling big data solutions on public cloud.
    • Experience with Unix/ Linux and shell scripting
    • Experience working with various data sources like unstructured data files, flat files, message queues, JSON based events, databases.
    • Experience driving architectural discussions and proposing solutions for system and product changes.
    • Ability to produce clear, well-documented designs for product features.
    • Experience writing high-performance, reliable and maintainable code.
    • Proven track record of strong analytical and problem-solving skills.
    • Ability to adapt to new development environments, changing business requirements.
    • Experience with LARGE DISTRIBUTED SERVICES IS A PLUS as is building/operating highly available systems • Proven ability to effectively work across multiple teams on solutions that have complex dependencies and requirements in a fast-paced environment

    COMMUNICATION SKILLS:

    • Able to effectively communicate across teams and roles
    • Excellent verbal and written communication skills Reasoning Ability
    • Proven ability as a problem-solver Other Skills
    • Strong estimating and planning skills
    • Ability and desire to thrive in a proactive, high-pressure, client-services environment

    #ZR