Reading:
Data Engineering Lead

5 months ago

Data Engineering Lead


We are looking for a Senior Data Engineer with a track record of building enterprise grade data platform for cloud products. This is an exciting opportunity for a technologist to build an innovative SaaS product using state-of-the-art and modern cloud technology stack.

As a Data Engineering Lead, you will be working on all aspects of data, from platform and infra build out to pipeline engineering and writing tooling/services for augmenting and fronting the core platform. The data platform interfaces with large systems to ingest data (batch, micro-batch) to create statistical/machine learning models in a distributed setting.

You will work closely with data scientists, product managers, legal, compliance and business stakeholders.

You will also evaluate new technologies/frameworks and contribute to various open source big data technologies. You are encouraged to think out of the box and have fun exploring the latest patterns and designs in the fields of software and data engineering.

Your responsibilities

  • Architecting and implementing a robust data platform for Stemly’s products. Design and develop extremely efficient, reliable and observable data pipelines using technologies such as Airflow, Kubernetes, ElasticSearch, Kafka/KubeMQ, Postgres, ClickHouse etc.
  • Design and deliver the next-gen data lifecycle management suite of tools/frameworks, including ingestion and consumption on the top of the data lake to support real-time, API-based and serverless use-cases, along with batch (mini/micro).
  • Build and expose metadata catalogue for the Data Lake for easy exploration, profiling as well as lineage requirements
  • Work on the development of the data platform iteratively by building quick POCs and converting ideas into real products.
  • Make data discoverable and easy to use for Data Scientists and Analysts. Enable Data Science teams to test and productize various ML models,
  • Develop code and mentor junior developers to ensure deliverables are on time, within budget, and with good code quality. Support your colleagues by reviewing code and designs.
  • Coordinate Continuous Integration activities, testing automation frameworks, and other related items in addition to contributing core product code.
  • Present technical solutions, capabilities, considerations, and features in business terms. Effectively communicate status, issues, and risks in a precise and timely manner.

About you

  • 8+ years of professional experience in software development in Python and Pandas.
  • Extensive experience with object-oriented design, coding and testing patterns, including experience with engineering software platforms and data infrastructures
  • Must have sound understanding for Big Data components & administration fundamentals
  • Hands-on in building a complete data platform using various open source technologies. Must have the knowledge to build metadata, lineage, observability and discoverability for data platform
  • Experienced in DevOps best practices like CI/CD, containerization, blue-green deployments, secrets management etc. in the Data ecosystem
  • Expert in the principles of distributed computing and data modelling
  • Expert in building optimized SQL queries
  • Conversant with the latest developments in the areas of Machine Learning models and efficiently supporting the data scientists is a plus
  • Experience with Airflow, ClickHouse, Postgres, Kubernetes, GraphDB is a big plus
  • Familiarity with Google Cloud Platform (e.g. GCS, Cloud Composer, BigQuery) is a plus
  • Outstanding communication and interpersonal skills

About us 

We are scientists and engineers with a strong passion for cutting-edge technology and decision science. Our mission is to enable enterprises to reduce time to decision with state-of-the-art forecasting and optimization. Our algorithms are based on automatic machine learning techniques and allow for the discovery of optimal forecasting models. We unlock sizable business and financial impact in terms of lower lost demand, decrease in inventory, lower working capital and financing costs. 

How to apply? 

You may already know if you’re a fit, but perhaps you’re worried about some of the requirements? We’re looking for smart and passionate Engineers; if you’re the right candidate, we’re flexible. 

Arrow-up