Design, optimize, and productionize scalable ETL/ELT pipelines using Databricks, PySpark, SQL, GCP, BigQuery, and Delta Lake. Build agentic workflows for data validation, troubleshooting, and automation. Develop reusable data engineering frameworks, implement quality and monitoring controls, optimize workloads, support deployments, and resolve complex production issues while collaborating with business, platform, and engineering teams.
We are looking for a Data Engineer with strong Databricks and GCP experience to build scalable ETL/ELT pipelines and productionize agentic workflows for data engineering automation.
The mandatory requirements are 5+ years of hands-on Databricks and PySpark experience, advanced SQL skills, strong GCP and BigQuery knowledge, and experience with Delta Lake and lakehouse architectures.
RECRUITMENT PROCESS
1. Application — Share a few details about your experience and background.
2. Coding Challenge — If applicable, complete it in the coding language you are most comfortable with.
3. Video Interview — Record a short video introduction in English.
4. Technical Interview or Hiring Manager Interview — Discuss your experience and fit for the role.
5. Offer — If it’s a match, you’ll receive an offer.
WHAT YOU’LL GAIN
- Remote work — Work from where you feel most productive.
- Local presence in India — Work in a structured and compliant environment aligned with Indian regulations.
- Competitive compensation in INR — Receive compensation in INR, plus support for learning, education, and wellness.
- Exciting projects — Work with modern technologies for global clients and fast-growing companies.
MUST HAVES
- 5+ years of strong hands-on experience with Databricks and PySpark.
- Advanced SQL and data-processing skills.
- Hands-on experience with GCP, particularly BigQuery.
- Experience with Delta Lake and modern data lake/lakehouse architectures.
- Strong understanding of ETL/ELT, data pipeline design, performance optimization, and data quality.
- Experience building reliable, scalable, production-grade data solutions.
- Strong analytical and troubleshooting skills.
- Understanding of software engineering practices, including testing, version control, deployment, monitoring, and production support.
- Upper-intermediate English level.
NICE TO HAVES
- Experience developing or integrating AI/agentic workflows, AI agents, or workflow automation solutions.
- Experience applying AI to automate data engineering, validation, troubleshooting, or operational processes.
- Familiarity with orchestration and automation frameworks.
- Experience developing reusable data engineering frameworks and platform components.
- Exposure to productionizing AI-enabled solutions with appropriate validation, monitoring, and human oversight.
WHAT YOU WILL DO
- Design, develop, and optimize scalable ETL/ELT pipelines using Databricks, PySpark, and SQL.
- Build and operationalize agentic workflows that automate data validation, issue identification, troubleshooting, and workflow execution.
- Integrate agentic capabilities with Databricks, GCP, BigQuery, and Delta Lake environments while supporting new business requirements and datasets.
- Build reusable frameworks and components that can support multiple data engineering and business use cases.
- Implement data quality checks, monitoring, validation, exception handling, and production controls.
- Optimize PySpark and SQL workloads and support testing, deployment, productionization, and ongoing enhancement of data and agentic solutions.
- Troubleshoot complex production issues and collaborate with business, data engineering, and platform teams to identify additional automation opportunities.
Similar Jobs
Software
Design, optimize, and productionize scalable ETL/ELT pipelines using Databricks, PySpark, SQL, GCP, BigQuery, and Delta Lake. Build agentic workflows for data validation, troubleshooting, and automation; develop reusable frameworks; implement quality checks, monitoring, exception handling, testing, deployment, and production support. Collaborate with business, data engineering, and platform teams to resolve complex production issues and identify automation opportunities.
Top Skills:
BigQueryDatabricksDelta LakeGoogle Cloud Platform (Gcp)PysparkSQL
Artificial Intelligence • Fintech • Information Technology • Logistics • Payments • Business Intelligence • Generative AI
Conduct red-team penetration testing of web, mobile, API, cloud, AI, and LLM applications. Assess vulnerabilities including prompt injection, jailbreaking, data poisoning, and model evasion; develop autonomous AI security-testing agents; perform remediation testing and reporting; and maintain application-security tools such as SAST, DAST, and WAF. Translate technical findings into business risks and collaborate cross-functionally to improve security.
Top Skills:
Ai/MlAPIsAWSAzureDastGCPLlmNistOsintOsstmmOwasp Top 10Owasp Top 10 For LlmsPtesPythonSans Top 25SastWaf
Artificial Intelligence • Healthtech • Professional Services • Analytics • Consulting
Lead consulting engagements that translate business requirements into technical solutions. Design and deliver data platforms, ETL pipelines, analytics solutions, architectures, prototypes, testing, deployment, and operations plans using Databricks, Snowflake, Python, PySpark, SQL, and AWS. Manage project teams, distribute tasks, mentor junior staff, communicate with clients, support project planning and risk mitigation, and ensure high-quality delivery across global teams.
Top Skills:
SparkAWSDatabricksETLGitPysparkPythonSnowflakeSQL
What you need to know about the Mumbai Tech Scene
From haggling for the best price at Chor Bazaar to the bustle of Crawford Market, the energy of Mumbai's traditional markets is a key part of the city's charm. And while these markets will always have their place, the city also boasts a thriving e-commerce scene, ranking among the largest in the region. Driven by online sales in everything from snacks to licensed sports merchandise to children's apparel, the local industry is worth billions, with companies actively recruiting to meet the demands of continued growth.
