Design, build, and operate scalable ETL/ELT pipelines using open-source tools. Optimize databases and SQL, prototype new tools, support migration from Azure Databricks to AWS, ensure data quality, observability, CI/CD, and collaborate across distributed teams in Advertising and Market Research.
We are hiring a hands-on Data Engineer to join our India-based team and help build the next generation of our data delivery platform. Working closely with our senior and principal engineers and with engineering leadership across North America and India, you will design, build, and operate the pipelines that move and transform millions of data points every day.
We are deliberately moving away from a locked-in, vendor-heavy stack toward a flexible, largely open-source architecture that keeps our options open. You will be in the engine room of that re-architecture — writing code, designing schemas, tuning queries, and helping prove out new tools before we adopt them at scale.
This is a build role. You bring strong hands-on data pipeline experience and the judgment to use the right open-source tool for the job, while taking technical direction on the broader platform strategy from our senior engineers. We expect you to get into the details and understand the “how” and “why” behind every pipeline.
What You’ll Do- Build and operate pipelines: Design, build, and optimize robust ETL/ELT pipelines that move and transform millions of daily data points reliably and efficiently.
- Work hands-on with the open-source stack: Build and maintain production data workflows using tools such as dbt, ClickHouse, and open-source orchestration frameworks (Airflow, Dagster, or similar).
- Engineer and tune databases: Write and optimize complex SQL across SQL Server and Postgres — schema design, indexing, performance tuning, query optimization, and root-cause analysis.
- Support platform modernization: Contribute to the hands-on migration away from Azure Databricks toward a more open, flexible, AWS-based stack, minimizing disruption to high-volume daily data delivery.
- Prototype and evaluate: Help build proofs-of-concept to benchmark new tools and patterns, and feed clear results back into the team’s adoption decisions.
- Work with AI-assisted tooling: Use AI coding assistants and well-crafted prompts (e.g., GitHub Copilot, Claude, ChatGPT) to accelerate pipeline development, SQL generation, debugging, testing, and documentation — always reviewing and validating the output before it ships.
- Safeguard data quality and reliability: Implement testing, validation, monitoring, observability, and CI/CD practices so data stays accurate and pipelines stay healthy at scale.
- Integrate across systems: Understand the systems upstream and downstream of your pipelines, from ingestion through to client-facing platforms, to ensure clean, end-to-end data delivery.
- Collaborate across a distributed team: Partner with engineers in North America and India, participate in code reviews, and learn the nuances of the Advertising and Market Research domain.
Requirements
- Experience: 7–8+ years in data engineering / ETL, with a strong track record as a hands-on engineer building and operating production data pipelines.
- Core databases: Strong SQL and RDBMS skills with solid, hands-on experience in SQL Server and Postgres (schema design, performance tuning, complex query optimization).
- Open-source and modern stack: Hands-on experience with tools such as dbt, ClickHouse, and open-source pipeline / orchestration tools (e.g., Airflow, Dagster), with the judgment to choose the right tool for the job.
- Programming: Strong proficiency in Python (or a similar language) for building and automating data pipelines.
- AI-assisted development: Hands-on experience using AI coding assistants and effective prompting techniques to work more efficiently, with the judgment to verify, test, and refine AI-generated code and queries.
- Cloud: Hands-on experience with a major cloud platform; AWS strongly preferred, as we are standardizing on AWS as we move off Azure Databricks.
- Engineering fundamentals: Comfortable with Git, code reviews, and writing tested, maintainable, well-documented code.
- Education: Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
- Data warehousing: Experience with cloud data warehouses or lakehouse patterns (Snowflake, BigQuery, Redshift, or similar).
- Streaming and real-time: Experience with streaming / real-time data technologies such as Kafka.
- Infrastructure: Familiarity with containerization (Docker, Kubernetes) and infrastructure-as-code (e.g., Terraform).
- Domain expertise: Previous experience in Advertising, Media, or Market Research.
Similar Jobs
Fintech • Professional Services • Consulting • Energy • Financial Services • Cybersecurity • Generative AI
Design, build, and maintain cloud-native data pipelines and ETL/ELT workflows for high-volume trade lifecycle data. Integrate DTCC sources (RTTM, Settlement Web) into a Unified Trade Record, process ISO 15022 messages, manage trade exception databases, optimize SQL Server performance, and ensure data quality, lineage, governance, and observability while collaborating with business stakeholders.
Top Skills:
AWSDockerDtcc RttmDtcc Settlement WebEksEltETLFixFxmlIbm MqIndexingIso 15022KafkaKubernetesRdsRest ApiS3SQLSQL ServerStored ProceduresUnified Trade Record (Utr)
Analytics
Build, optimize, and maintain scalable AWS-based data pipelines and data lakes (Snowflake/Databricks). Implement streaming ingestion (Kinesis/Kafka), CDC, automated data quality checks, and ML-ready feature stores. Collaborate with Product, ML, and Analytics teams, own pipeline QA, observability, and participate in on-call rotations to resolve production data issues.
Top Skills:
Apache IcebergAWSAws BedrockAws GlueAws KinesisAws S3Aws SnsAws SqsAws Step FunctionsDatabricksDatabricks AiDatabricks WorkflowsDebeziumDelta LakeFeature StoreFivetranKafkaParquetPysparkPythonSnowflakeSnowflake CortexSQL
eCommerce • Other • Retail
Design, build, and maintain enterprise ETL/ELT pipelines in Microsoft Fabric; ingest and transform data from diverse sources into dimensional star schemas; optimize performance and cost; implement data quality checks and monitoring; document architecture and pipelines; collaborate with analysts, stakeholders, and global engineering teams to deliver Power BI-ready analytics datasets.
Top Skills:
Advertising PlatformsAmazon Seller CentralAmazon Vendor CentralAPIsAzure Data FactoryBusiness CentralDataflow Gen2Fabric LakehouseFabric WarehouseJSONMicrosoft Dynamics NavMicrosoft FabricPower BIPythonQuickbooksRestSalesforce Commerce CloudShopifySparkSQLWebhooks
What you need to know about the Mumbai Tech Scene
From haggling for the best price at Chor Bazaar to the bustle of Crawford Market, the energy of Mumbai's traditional markets is a key part of the city's charm. And while these markets will always have their place, the city also boasts a thriving e-commerce scene, ranking among the largest in the region. Driven by online sales in everything from snacks to licensed sports merchandise to children's apparel, the local industry is worth billions, with companies actively recruiting to meet the demands of continued growth.
