Design and optimize data pipelines using Azure Databricks and Spark, ensuring data reliability and quality through ETL/ELT workflows and CI/CD automation.
Key Responsibilities Design, develop, and optimize data pipelines using Azure Databricks. Implement ETL/ELT workflows leveraging Spark (PySpark/Scala). Integrate and manage data from multiple sources such as Azure Data Lake, SQL DB, Synapse, and external APIs. Design and enforce data quality rules, validation checks, and monitoring frameworks. Work with data quality tools to profile, cleanse, and validate data. Ensure data pipelines follow best practices for performance, scalability, and cost optimization. Collaborate with data engineers, analysts, and business stakeholders to ensure data reliability. Implement CI/CD pipelines and automation for Databricks deployments. Maintain documentation for data processes, pipelines, and quality rules.
Hexaware Technologies Navi Mumbai, Maharashtra, IND Office
#1, Sector III, Millennium Business Park,, 'A' Block , TTC Industrial Estate, Mahape, Navi Mumbai, Maharashtra , India, 400710
Similar Jobs
Information Technology • Consulting
Design, build, and optimize scalable data pipelines and analytics solutions on Snowflake. Collaborate with teams for data products and enforce security and governance.
Top Skills:
SnowflakeSQL
Information Technology • Consulting
Design and optimize scalable data pipelines and analytics platforms using Snowflake, Airflow, and SQL. Ensure data quality and drive performance optimization.
Top Skills:
AirflowAWSAzureDbtDebeziumDockerFivetranGCPGitKafkaKinesisKubernetesPub/SubPythonSnowflakeSQLTerraform
Information Technology • Consulting
The role requires leading big data projects with hands-on experience in Azure Databricks, Pyspark coding, advanced SQL, and data warehouse management.
Top Skills:
Azure DatabricksPysparkPythonSQL
What you need to know about the Mumbai Tech Scene
From haggling for the best price at Chor Bazaar to the bustle of Crawford Market, the energy of Mumbai's traditional markets is a key part of the city's charm. And while these markets will always have their place, the city also boasts a thriving e-commerce scene, ranking among the largest in the region. Driven by online sales in everything from snacks to licensed sports merchandise to children's apparel, the local industry is worth billions, with companies actively recruiting to meet the demands of continued growth.
