Lead design and implement enterprise-scale data solutions on Azure Databricks and related services. Architect end-to-end ETL/streaming pipelines, optimize Spark/SQL workloads, automate CI/CD and testing, enforce data governance (GxP, HIPAA/GDPR), enable self-service analytics with Power BI/Tableau, and provide technical leadership and stakeholder alignment.
The Lead Data Engineer will design, build, and operationalize scalable data solutions to support enterprise analytics and AI/ML initiatives. This role requires expert-level proficiency in Databricks, Azure Fabric, PySpark, SQL, and the Azure ecosystem, with deep experience across data warehouses, data lakes, and real-time integration. The Lead Data Engineer will architect end-to-end pipelines using industry-standard tools, drive automation, and move solutions effectively into production. The incumbent will ensure compliance with data governance requirements (including GxP and HIPAA/GDPR) while building reusable, integrated pipelines and analytical models that promote self-service analytics. This role provides technical leadership across the team, mentors junior engineers, and partners with business stakeholders to align data engineering with organizational objectives.
About the Role
- Architect, design, and implement end-to-end data solutions using Azure Databricks, PySpark, Azure Data Factory, and Azure SQL.
- Design, build, and maintain data pipelines from data sources through integration to consumption for specific use cases.
- Implement robust data modeling standards across bronze, silver, and gold layers in the data lake.
- Develop data models (conceptual, logical, and/or physical) as required.
- Optimize Spark and SQL workloads for performance, scalability, and cost efficiency.
- Manage metadata using data preparation, integration, and AI-enabled tools and techniques.
- Drive automation in data integration; recommend and lead implementation of techniques to automate repeatable data preparation and integration tasks.
- Build API-based integrations (REST/JSON) and real-time ingestion frameworks.
- Automate data workflows using Azure DevOps pipelines and Git-based CI/CD practices.
- Implement parameterized, reusable pipeline templates for ingestion and transformation.
- Develop automated unit, regression, and integration testing frameworks for data jobs.
- Prepare and curate high-quality datasets for BI, reporting, and advanced analytics.
- Partner with analytics teams using Power BI, Tableau, or similar platforms to define semantic models and KPIs.
- Implement performance-optimized data models for self-service analytics.
- Will occasionally provide support to end users on the use of data visualization solutions.
- Lead technical design reviews, mentor junior engineers, and promote best practices.
- Assist cross-functional groups, business analysts, and stakeholders to gather, define, and refine data requirements.
- Collaborate with business and IT stakeholders to align data engineering with organizational objectives.
- Propose innovative data ingestion, preparation, and integration techniques to address stakeholder requirements.
- Contribute to architectural roadmaps and technology evaluations for the data platform.
- In collaboration with functional leaders, identify inefficiencies and recommend improvements to the executive team.
. Data Architecture & Engineering
. Data Integration & Automation
. Analytics & Data Enablement
Stakeholder Engagement & Leadership
About You
- Expert-level proficiency in Databricks, Azure Fabric, PySpark, SQL, and Azure DevOps.
- Proven experience with Azure Data Factory, ADLS Gen2, and Azure SQL Server.
- Strong experience with Microsoft Azure data management architectures including Data Warehouse, Data Lake, and Data Catalogue, and supporting processes such as Data Integration, Governance, and Metadata Management.
- Experience with Power BI required; Tableau or Looker a plus.
- Working knowledge of CI/CD automation, version control (Git), and infrastructure as code (ARM, Bicep, or Terraform).
- Experience in life sciences or healthcare industries is a strong plus.
- Good understanding of GxP, GDPR/HIPAA, and applicable CFR/CTR/CTD regulations.
- Demonstrated success working with both IT and business stakeholders while integrating analytics and data science output into business processes and workflows.
- Must have excellent written and verbal communication skills.
- Proven ability to work independently and as part of a team and meet important deadlines.
- Statistical analysis skills are an asset.
Job Experience & Education Requirements:
Bachelor’s degree in Computer Science, Information Systems, Engineering, or related field (Master’s preferred)
And
5–8 years of experience designing and developing enterprise-scale data solutions (data warehouses, data lakes, operational databases)
Other:
Similar Jobs
Fintech • Professional Services • Consulting • Energy • Financial Services • Cybersecurity • Generative AI
Design, build, and maintain scalable cloud-native data pipelines for trade lifecycle processing. Ingest, transform, and normalize high-volume financial data (DTCC, ISO15022), implement MQ/API-based ingestion, optimize SQL Server performance, maintain UTR schemas and trade exception databases, ensure data quality, lineage and governance, and collaborate with reconciliation, settlements, and operations stakeholders.
Top Skills:
Aws EksAws RdsAws S3DockerDtcc RttmDtcc Settlement WebEtl/Elt FrameworksFixFxmlIbm MqIso 15022KafkaKubernetesRest ApisSQL ServerUnified Trade Record (Utr)
Healthtech • Pharmaceutical • Manufacturing
Design, build, and maintain scalable ETL/ELT, batch and streaming pipelines on cloud platforms; support embedding and vector-based pipelines; create semantic data models; implement data quality, lineage, and monitoring; enable offline training and low-latency inference; collaborate with stakeholders across domains and mentor other data engineers.
Top Skills:
AWSAws GlueAzureAzure Data FactoryCsvDatabricksDatabricks WorkflowsFeature StoresJSONNoSQLPandasParquetPolarsPythonSnowflakeSparkSQLVector DatabasesXML
Software
Design, build, monitor and scale an enterprise data warehouse and pipelines. Lead data integration, ETL design, source-to-target mapping, reusable frameworks, automated testing/CI, and partner with stakeholders to deliver secure, governed analytics for product, marketing and HR use cases.
Top Skills:
Adp WfnAPIsCi/CdETLFivetranHubspotJSONPythonRedshiftSalesforceSnowflakeSQLStitchXML
What you need to know about the Mumbai Tech Scene
From haggling for the best price at Chor Bazaar to the bustle of Crawford Market, the energy of Mumbai's traditional markets is a key part of the city's charm. And while these markets will always have their place, the city also boasts a thriving e-commerce scene, ranking among the largest in the region. Driven by online sales in everything from snacks to licensed sports merchandise to children's apparel, the local industry is worth billions, with companies actively recruiting to meet the demands of continued growth.
