Leads the design, development, and maintenance of scalable ETL/ELT pipelines and data products across distributed, multi-cloud environments. Defines data engineering architectures, resolves complex production issues, and designs efficient data solutions. Provides technical leadership and mentorship while collaborating with architects, engineers, and business stakeholders. Requires expertise in big data, AWS services, data warehousing, databases, data modeling, orchestration, streaming, security, observability, version control, containers, and modern development practices.
Job Title Lead Data Engineer Location (s) Cochin Years of Experience 5-10yrs Job Description
Role & Responsibilities Technical Qualifications Python + AWS EMR + AWS Glue + Snowflake + ETL/ELT + Data Engineering + Data Architecture + Big Data + SQL
Lead Data Engineer
What You’ll Do
- Lead the design, development, and maintenance of scalable ETL/ELT pipelines and data products in multi-cloud, multi-region, distributed environments.
- Define and apply appropriate data engineering patterns and architectures based on business and technical requirements.
- Drive technical investigations and resolve complex data, operational, and production issues.
- Design scalable, flexible, efficient, and supportable solutions using appropriate technologies and disciplined development practices.
- Provide technical leadership, guidance, and mentorship to data engineering teams.
- Collaborate with architects, engineering teams, and business stakeholders to deliver robust data solutions.
Key Skills
- Data Engineering & Architecture: Strong understanding of data engineering patterns and modern architectures such as Data Mesh, Data Fabric, Data Lake, and Data Warehouse.
- Big Data & Cloud: Proficiency in Amazon EMR, AWS Glue, Data Lake, and technologies for processing large datasets.
- Programming: Strong proficiency in Python, Java, or Scala, with solid OOP expertise.
- Data Warehousing: Strong knowledge of data warehousing solutions, preferably Snowflake.
- ETL/ELT & Orchestration: Hands-on experience with ETL/ELT pipelines and data orchestration tools.
- Databases: Strong expertise in relational and NoSQL databases.
- Data Modelling: Experience designing efficient OLAP and OLTP data models.
- Streaming: Knowledge of streaming data technologies and architectures.
- Version Control: Experience with Git and modern development practices.
- Containers & Orchestration: Understanding of Docker and Kubernetes.
- Data Security: Knowledge of encryption, access control, data privacy, and compliance.
- Monitoring & Logging: Experience implementing monitoring, logging, alerting, and observability for data pipelines.
Role & Responsibilities
- Lead Data Engineer
- What You’ll Do
- Lead the design, development, and maintenance of scalable ETL/ELT pipelines and data products in multi-cloud, multi-region, distributed environments.
- Define and apply appropriate data engineering patterns and architectures based on business and technical requirements.
- Drive technical investigations and resolve complex data, operational, and production issues.
- Design scalable, flexible, efficient, and supportable solutions using appropriate technologies and disciplined development practices.
- Provide technical leadership, guidance, and mentorship to data engineering teams.
- Collaborate with architects, engineering teams, and business stakeholders to deliver robust data solutions.
- Key Skills
- Data Engineering & Architecture: Strong understanding of data engineering patterns and modern architectures such as Data Mesh, Data Fabric, Data Lake, and Data Warehouse.
- Big Data & Cloud: Proficiency in Amazon EMR, AWS Glue, Data Lake, and technologies for processing large datasets.
- Programming: Strong proficiency in Python, Java, or Scala, with solid OOP expertise.
- Data Warehousing: Strong knowledge of data warehousing solutions, preferably Snowflake.
- ETL/ELT & Orchestration: Hands-on experience with ETL/ELT pipelines and data orchestration tools.
- Databases: Strong expertise in relational and NoSQL databases.
- Data Modelling: Experience designing efficient OLAP and OLTP data models.
- Streaming: Knowledge of streaming data technologies and architectures.
- Version Control: Experience with Git and modern development practices.
- Containers & Orchestration: Understanding of Docker and Kubernetes.
- Data Security: Knowledge of encryption, access control, data privacy, and compliance.
- Monitoring & Logging: Experience implementing monitoring, logging, alerting, and observability for data pipelines.
Similar Jobs
Cloud • Information Technology • Productivity • Security • Software • App development • Automation
Lead the technical evolution of Atlassian’s data engineering capabilities by architecting scalable batch and streaming data solutions. Direct complex engineering efforts, improve data quality and productivity, establish technical direction, and coordinate implementation across teams. Mentor and coach data engineers while partnering cross-functionally to support data-driven business decisions and long-term platform scalability.
Top Skills:
Apache AirflowApache FlinkApache HiveApache KafkaSparkAws AthenaAws EmrAws RedshiftDatabricksKappa ArchitectureLambda ArchitectureMaster Data ManagementMicroservicesSQL
Healthtech • Pharmaceutical • Manufacturing
Designs, develops, and maintains scalable ETL and data integration pipelines using Snowflake, AWS Glue, Fivetran, and Snowpipes. Responsibilities include data transformation, quality validation, performance tuning, monitoring, troubleshooting, documentation, and data lineage. The role collaborates with architects, modelers, and source-system owners to integrate databases, APIs, files, and other data sources. It requires advanced SQL, scripting, relational and NoSQL database knowledge, and substantial ETL engineering experience.
Top Skills:
APIsAws GlueAws S3Azure Data FactoryChange Data CaptureCopy CommandDatastageFivetranGoogle DataflowHadoopHvrInformatica IicsInformatica PowercenterNoSQLOraclePerlPostgresPythonSftpShell ScriptingSnowflakeSnowpipeSparkSQLSQL Server
.jpg)
Fintech • Analytics
Oversee and improve cloud consumption models, data inputs, reporting, forecasting, and scenario analysis for Azure application migrations. Maintain accurate data stores, validate delivery assumptions, identify trends and gaps, optimize cloud consumption, and provide recommendations to improve transparency, root cause analysis, and decision-making across technology and product teams.
Top Skills:
DatabricksAzureExcelPower BIPython
What you need to know about the Mumbai Tech Scene
From haggling for the best price at Chor Bazaar to the bustle of Crawford Market, the energy of Mumbai's traditional markets is a key part of the city's charm. And while these markets will always have their place, the city also boasts a thriving e-commerce scene, ranking among the largest in the region. Driven by online sales in everything from snacks to licensed sports merchandise to children's apparel, the local industry is worth billions, with companies actively recruiting to meet the demands of continued growth.
