Design, build, and maintain scalable ETL/ELT pipelines and lakehouse architectures on Azure Databricks. Implement batch and streaming workflows, enforce data modeling and governance, optimize Spark transformations, deploy using CI/CD and IaC, mentor junior engineers, and collaborate with data scientists to productionize ML models.
JOB DESCRIPTION
Senior Data Engineer
Data & Analytics Engineering
You will collaborate closely with data scientists, analysts, and product teams to deliver high-quality, reliable, and performant data solutions in a fast-paced environment.
Senior Data Engineer
Data & Analytics Engineering
About the Role
We are looking for a Senior Data Engineer to join our growing Data & Analytics team. In this role, you will design, build, and maintain scalable data pipelines and platforms that power critical business decisions. You will be a key contributor in driving our cloud-first data strategy, leveraging Azure Databricks as a core technology and Azure Data Factory (ADF) as Good to have feature.You will collaborate closely with data scientists, analysts, and product teams to deliver high-quality, reliable, and performant data solutions in a fast-paced environment.
Key Responsibilities
Data Pipeline & Architecture- Design, develop, and maintain robust ETL/ELT pipelines using Azure Databricks and Azure Data Factory (ADF).
- Architect and implement scalable data lakehouse solutions on Azure using Delta Lake.
- Build and optimize data workflows across batch and streaming workloads (Spark Structured Streaming, Event Hubs).
- Define and enforce data modeling best practices (star schema, data vault, medallion architecture).
- Develop and optimize Spark-based data transformations using PySpark and Spark SQL in Databricks.
- Manage Databricks clusters, jobs, and workspace configurations for performance and cost efficiency.
- Implement Delta Live Tables (DLT) pipelines for declarative, auto-scaling data transformations.
- Leverage Unity Catalog for data governance, lineage tracking, and access control.
- Utilize Databricks Asset Bundles (DABs) and CI/CD practices for deployment automation.
- Build, schedule, and monitor complex ADF pipelines with parameterized templates.
- Integrate ADF with Azure Key Vault, Linked Services, and Integration Runtimes (SHIR/Azure IR).
- Implement incremental load patterns, watermarking, and change data capture (CDC) strategies.
- Troubleshoot pipeline failures and optimize ADF pipeline performance and cost.
- Implement data quality frameworks and validation checks across pipelines.
- Enforce data cataloging, lineage, and metadata management practices.
- Collaborate with data governance teams to ensure compliance with data policies and regulations (GDPR, HIPAA).
- Mentor junior data engineers and conduct code reviews.
- Work closely with data scientists and ML engineers to productionize machine learning models.
- Partner with DevOps/Cloud teams on infrastructure-as-code (Terraform/Bicep) for data platform provisioning.
- Document architecture decisions, pipeline designs, and operational runbooks.
Required Qualifications
- 7+ years of hands-on experience with Azure Databricks (Spark SQL, Delta Lake, PySpark).
- Strong proficiency in SQL preferred and Python.
- Deep understanding of distributed computing principles and the Apache Spark ecosystem.
- Experience with Azure data services: ADLS Gen2, Azure Synapse, Azure SQL, Event Hubs / Kafka.
- Solid understanding of data warehousing concepts and dimensional modeling.
- Experience with version control (Git) and CI/CD tools (Azure DevOps, GitHub Actions).
- Familiarity with infrastructure-as-code tools (Terraform or ARM/Bicep).
Preferred Qualifications
- Databricks Certified Data Engineer Associate or Professional certification.
- Microsoft Certified: Azure Data Engineer Associate (DP-203).
- Experience with Delta Live Tables (DLT) and Databricks Workflows.
- Familiarity with Power BI or other BI/reporting tools.
- Experience with Scala or Java for Spark development.
Technology Stack
Core PlatformsAzure Databricks (Preferred), Azure Data Factory (ADF) (Good to have)Data StorageAzure Data Lake Storage Gen2, Delta Lake, Azure SQL, Cosmos DBLanguagesPython (PySpark), SQLOrchestrationADF Triggers, Databricks Workflows, Apache AirflowMonitoringAzure Monitor, Log Analytics, Databricks Cluster PoliciesGovernanceUnity Catalog, Azure Purview, Azure Key Vault – (Good to have)BI / ReportingPower BI, Tableau Key Competencies
- Strong analytical and problem-solving skills with attention to detail.
- Excellent communication skills — ability to translate complex technical concepts to non-technical stakeholders.
- Self-motivated and able to manage competing priorities in an agile environment.
- Collaborative team player with a growth mindset and eagerness to mentor others.
- Proactive in identifying performance bottlenecks and proposing architectural improvements.
Similar Jobs
Artificial Intelligence • Consumer Web • Digital Media • Information Technology • Social Impact • Software
Build end-to-end full-stack AI features on a Ruby on Rails backend and React frontend, design and run experiments (A/B tests and AI evaluations), implement AI infrastructure for reliable LLM inference, and iterate quickly to scale production-ready LLM-powered products.
Top Skills:
Ai AgentsLlmsMySQLPostgresReactRetrieval-Augmented Generation (Rag)Ruby On Rails
Financial Services
Design, build, test, and ship production AI-enabled software and reusable components for enterprise workflows. Implement context-aware agentic systems, integrate models and APIs, write unit tests, debug production issues, lead code reviews, mentor engineers, participate in on-call rotation, and collaborate with U.S. business partners while working remotely from India.
Top Skills:
Agentic SystemsAPIsCi/CdData ModelingLlmMessagingObservabilityReliabilityVersion Control
Other • Retail
Design, build, and maintain scalable Agentic AI platforms and services. Develop SDKs, APIs, and microservices; implement data governance, observability, and MLOps best practices. Work cross-functionally, participate in code reviews, mentor junior engineers, and deliver production-quality AI/LLM enterprise applications.
Top Skills:
Agent EvalAngularApache AirflowSparkAws CloudformationAws RdsAws SagemakerAzure MlAzure SqlBedrockClaude CodeCursor.AiCypressDatabricksGCPGcp Cloud SqlGitGoJavaLangchainLanggraphLintersLlmsMlopsModel Context Protocol (Mcp)NoSQLPytestPythonRag PipelinesRdbmsReactScalaScss/SassSnowflakeTerraformType CheckersTypescriptVue
What you need to know about the Mumbai Tech Scene
From haggling for the best price at Chor Bazaar to the bustle of Crawford Market, the energy of Mumbai's traditional markets is a key part of the city's charm. And while these markets will always have their place, the city also boasts a thriving e-commerce scene, ranking among the largest in the region. Driven by online sales in everything from snacks to licensed sports merchandise to children's apparel, the local industry is worth billions, with companies actively recruiting to meet the demands of continued growth.
