Designs, develops, and governs enterprise data platforms using Databricks and Microsoft Fabric. Builds scalable pipelines, lakehouses, warehouses, semantic models, data marts, automation frameworks, and data-quality validation systems. Leads data engineering through the full delivery lifecycle, implements security and governance controls, supports CI/CD and infrastructure automation, optimizes Spark workloads, and collaborates with stakeholders and technical teams. The role also includes mentoring engineers and establishing reusable engineering standards.
Title: Senior Data Engineer – Databricks & Fabric Engineering
Location: Hyderabad / Mumbai / Bangalore
Education: Bachelor's Degree
Experience: 5+ Years
We are seeking an experienced Senior Data Engineer with strong expertise in Databricks, Microsoft Fabric, Data Engineering, Automation, and Data Quality Engineering. The ideal candidate will play a key role in designing, building, automating, and governing enterprise data platforms while ensuring the reliability, accuracy, and security of analytics solutions.
The role will support strategic data modernization initiatives across multiple clients and industry domains, focusing on scalable data pipelines, data quality frameworks, automation, and analytics platform engineering.
Key Responsibilities- Design, develop, and maintain scalable data solutions using Databricks, Microsoft Fabric, SQL, Spark, and Delta Lake.
- Build and optimize data ingestion, transformation, and orchestration pipelines for enterprise data platforms.
- Develop and implement automation frameworks for data validation, testing, monitoring, deployment, and operational support.
- Lead data engineering activities across the full lifecycle, including requirements, design, development, testing, deployment, and support.
- Build and maintain automated QA and reconciliation frameworks for analytical and reporting solutions.
- Validate data quality, completeness, consistency, and business rule compliance across enterprise datasets.
- Develop and maintain semantic models, lakehouses, warehouses, and data marts within Microsoft Fabric.
- Implement and validate security controls, including Row-Level Security (RLS) and data governance standards.
- Collaborate with business stakeholders, architects, analytics teams, and platform engineers to deliver enterprise-grade solutions.
- Support CI/CD, DevOps, Infrastructure-as-Code, and deployment automation initiatives.
- Drive performance optimization and cost efficiency across Databricks and Fabric environments.
- Establish engineering best practices, coding standards, reusable frameworks, and governance processes.
- 5+ years of experience in Data Engineering, Analytics Engineering, or Data Platform Development.
- Strong hands-on expertise in:
- Databricks
- Apache Spark (PySpark)
- SQL
- Microsoft Fabric
- Delta Lake
- Data Warehousing
- Experience designing and implementing:
- Data Pipelines
- ETL/ELT Solutions
- Lakehouse Architectures
- Data Integration Frameworks
- Strong experience with Databricks Workflows, Jobs, Automation, and Orchestration.
- Experience developing automated testing and validation frameworks for data platforms.
- Hands-on experience implementing CI/CD pipelines using Azure DevOps, GitHub, or equivalent tools.
- Strong understanding of data quality, reconciliation, metadata management, and governance practices.
- Experience with performance tuning and optimization of Spark workloads.
- Experience working with Microsoft Fabric components, including:
- Data Factory
- Lakehouse
- Warehouse
- Semantic Models
- Power BI
- Experience with Infrastructure as Code (Terraform, Bicep, ARM Templates).
- Experience with DataOps and MLOps processes.
- Knowledge of Power BI semantic models, DAX, and analytical reporting solutions.
- Experience with enterprise data governance and security frameworks.
- Familiarity with cloud platforms such as Azure, AWS, or GCP.
- Experience leading technical discussions, mentoring engineers, and driving engineering best practices.
- Strong analytical and problem-solving skills.
- Excellent communication and stakeholder management abilities.
- Ability to lead initiatives independently and work effectively in cross-functional teams.
- Strategic mindset with the ability to align technical solutions with business objectives.
Datavail’s Data Management Services:
Datavail’s Data Management and Analytics practice is made up of experts who provide a variety of data services including initial consulting and development, designing and building complete data systems, as well as ongoing support and management of database, data warehouse, data lake, data integration, and virtualization and reporting environments. Datavail’s team is comprised of not just excellent BI & analytics consultants, but great people as well. Datavail’s data intelligence consultants are experienced, knowledgeable and certified in the best in breed BI and analytics software applications and technologies. We ascertain your business objectives, goals and requirements, assess your environment, and recommend the tools which best fit your unique situation. Our proven methodology can help your project succeed, regardless of stage. With the combination of a proven delivery model and top-notch experience ensures that Datavail will remain the Data Management experts on demand you desire. Datavail’s flexible and client focused services always add value to your organization.
Similar Jobs
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Develop, test, validate, and support large-scale batch and real-time data platforms using Spark, Kafka, and NiFi. Build resilient data pipelines, validation frameworks, reusable platform components, and deployment automation. Optimize distributed systems for performance, scalability, and reliability; support production operations, monitoring, troubleshooting, and incident resolution. Collaborate on architecture, CI/CD, documentation, and engineering standards while mentoring junior engineers.
Top Skills:
Apache KafkaApache NifiApache OzoneSparkCephCi/CdDockerGitHdfsKubernetesLinuxPysparkPythonScalaShell ScriptingSQL
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Designs, develops, and supports scalable cloud data pipelines, transformations, data models, and reporting datasets. Partners with engineering, business intelligence, and business teams to deliver reliable, governed data solutions. Monitors and troubleshoots production processes, improves performance and reliability, contributes to data quality and documentation standards, and participates in code reviews and engineering best practices.
Top Skills:
SparkDatabricksDbtSnowflakeSQL
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Build data engineering solutions that ingest, aggregate, unify, and analyze high-volume data from diverse sources. Develop data pipelines and models using Cloudera, Databricks, AWS, Hadoop, Python, and SQL. Partner with product, data science, platform, technology, client, and sales teams to deliver actionable insights, prototypes, business intelligence, and innovative cyber, fraud, security, and digital identity products.
Top Skills:
Apache NifiAWSAzure Data FactoryCloudera AiCloudera Data EngineeringDatabricksGraph DatabaseHadoopImpalaNumpyPandasPentahoPysparkPythonSQLTalend
What you need to know about the Mumbai Tech Scene
From haggling for the best price at Chor Bazaar to the bustle of Crawford Market, the energy of Mumbai's traditional markets is a key part of the city's charm. And while these markets will always have their place, the city also boasts a thriving e-commerce scene, ranking among the largest in the region. Driven by online sales in everything from snacks to licensed sports merchandise to children's apparel, the local industry is worth billions, with companies actively recruiting to meet the demands of continued growth.
