Design, build, and optimize end-to-end data pipelines for large structured and unstructured data. Implement near-real-time ETL, data validation, monitoring, and performance optimization. Collaborate with stakeholders, document designs and workflows, and provide technical guidance to the team.
Location: Pune
Responsibilities include:- Design, implement, and optimize end-to-end data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.
- Develop data pipelines to extract and transform data in near real-time using cloud-native technologies.
- Implement data validation and quality checks to ensure accuracy and consistency.
- Monitor system performance, troubleshoot issues, and implement optimizations to enhance reliability and efficiency.
- Collaborate with business users, analysts, and other stakeholders to understand data requirements and deliver tailored solutions.
- Document technical designs, workflows, and best practices to facilitate knowledge sharing and maintain system documentation.
- Provide technical guidance and support to team members and stakeholders as needed.
- 8+ years of work experience.
- Proficiency in writing complex SQL queries on MPP systems (Snowflake/Redshift).
- Experience in Databricks and Delta tables.
- Data engineering experience with Spark/Scala/Python.
- Experience in Microsoft Azure stack (Azure Storage Accounts, Data Factory, and Databricks).
- Experience in Azure DevOps and CI/CD pipelines.
- Working knowledge of Python.
- Comfortable participating in 2-week sprint development cycles.
Similar Jobs
Agency • Information Technology
Design, build, optimize, and maintain high-performance Spark-based data pipelines using Scala/Java and Hive on Hadoop/CDP. Own full project lifecycle, enforce coding best practices, troubleshoot Spark/Hive/YARN performance, and collaborate with stakeholders to deliver scalable data solutions.
Top Skills:
SparkCloudera Data Platform (Cdp)HadoopHiveJavaScalaYarn
Digital Media • Information Technology • News + Entertainment
Design, develop, and maintain microservices-based Python applications and RESTful APIs. Implement asynchronous processing, ensure test coverage and performance, support CI/CD and containerized deployments, mentor junior engineers, and collaborate with cross-functional teams to deliver scalable, reliable software.
Top Skills:
CeleryDjangoDjango Rest FrameworkDockerFastapiGitGitlabGitlab CiJenkinsKubernetesMongoDBNoSQLPostgresPythonRabbitMQRedisRestful ApisSQL
Cloud • Information Technology • Security • Software • Cybersecurity
Drive strategic enterprise sales into named accounts across India (Bengaluru/Delhi). Proactively hunt, plan accounts, build senior relationships, manage complex sales cycles and contract negotiations, maintain pipeline, and close deals for Cloudflare network security, CDN, and internet solutions. Up to 50% travel.
Top Skills:
CdnCloudflareComputer NetworkingDnsNetwork SecuritySaaSZero Trust
What you need to know about the Mumbai Tech Scene
From haggling for the best price at Chor Bazaar to the bustle of Crawford Market, the energy of Mumbai's traditional markets is a key part of the city's charm. And while these markets will always have their place, the city also boasts a thriving e-commerce scene, ranking among the largest in the region. Driven by online sales in everything from snacks to licensed sports merchandise to children's apparel, the local industry is worth billions, with companies actively recruiting to meet the demands of continued growth.
