Lead and manage a team of Data Engineers, focusing on ETL processes, data integrity, and AWS services. Responsible for technical leadership and operational management.
Embark on an exciting journey into the realm of software product development with 3Pillar! We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a Lead, Data Engineering, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.
Remote Opening
Shift: Day Shift (9-hour shift, 5 days a week, weekends and holidays on rotation).
This role requires covering weekends and holidays on a rotational basis to ensure 7-day-a-week coverage.
The critical time window to be covered is 11:00 AM to 7:00 PM IST
Responsibilities:
- Team & Operational Management
- Team Management: Manage the on-site team of 3-4 Data Engineers.
- Roster & Scheduling: Create, manage, and maintain team rosters and schedules to ensure full 8-hour coverage, 7 days a week, including rotational weekend and holiday coverage.
- Availability: Must be able to work a day shift and manage a team that provides 7-day-a-week coverage, including weekends and holidays on rotation.
- Coverage Assurance: Handle all team leave requests (planned and unplanned). You are responsible for ensuring coverage is always maintained, especially during medical or other unexpected absences.
- Performance & SLAs: Act as the primary operational point of contact, ensuring the team executes all tasks efficiently, makes sound decisions, and adheres to all agreed-upon SLAs.
- Technical Leadership & Engineering
- Final Escalation Point: Serve as the final technical escalation point for the team; personally handle the most critical job failures and system issues.
- Expert RCA: Lead all high-priority Root Cause Analysis (RCA) efforts, documenting solutions and implementing permanent fixes to prevent recurrence.
- Strategic Improvement: Drive the team's efforts in performance tuning and AWS Glue cost optimization.
- Automation Strategy: Define and lead the development of new automation jobs (using Glue, Lambda, and Step Functions) to reduce manual support tasks.
- Complex Data Management: Oversee all data integrity and synchronization tasks, including CDC reloads and write-backs to external databases like Teradata.
- Knowledge Transfer: Own the knowledge transfer process to internal resources and ensure the team maintains excellent documentation and runbooks.
Qualification:
- Demonstrated expertise with a minimum of 8 years of relevant experience in data engineering with experience of leading and managing a technical support or data engineering team.
- Ability to function as a player-coach — leading while also contributing hands-on.
- Proficiency in Python for data engineering, ETL/ELT workflows, and automation tasks.
- Solid understanding of data governance, security, and performance optimization practices.
- Experience designing, creating and maintaining data pipelines in AWS environments, with strong exposure to AWS Data services like S3, Glue, Lambda, and Step Functions, and other related services.
- Data Architecture: Expert-level understanding of ETL, data warehouse design, and pipeline optimization.
- Database Expertise: Deep experience with Teradata, Change Data Capture (CDC) processes, and data synchronization.
- Integration: Experience with Lambda-based integrations, such as triggering SAS or RMJ events.
- Strong communication skills and ability to translate complex technical concepts for business stakeholders.
- Resilience: Ability to make critical, autonomous decisions and lead a team in a high-pressure environment.
- Preferred Skills: AWS Certified Solutions Architect
Top Skills
Aws Glue
Aws Lambda
Data Governance
ETL
Python
Step Functions
Teradata
Similar Jobs
Artificial Intelligence • Digital Media • Marketing Tech • Software
As a Lead Data Engineer at Adobe, you will create data platforms, build data pipelines, collaborate with teams, and mentor junior engineers, driving innovation and delivering high-quality solutions.
Top Skills:
AirflowSparkAzure DatabricksDatahubDelta LakeJavaOoziePython
Machine Learning • Natural Language Processing
The Machine Learning R&D Engineer designs, develops, and implements machine learning solutions, manages projects, optimizes models, and mentors junior members.
Top Skills:
Aws Ec2Aws S3DockerPythonPyTorchScikit-LearnTensorFlow
Artificial Intelligence • Healthtech • Professional Services • Analytics • Consulting
Develop and apply advanced statistical models, use data to guide decisions, and communicate results to clients while mentoring Associates.
Top Skills:
ExcelHadoopJavaPythonRSASTableauVisual Basic
What you need to know about the Mumbai Tech Scene
From haggling for the best price at Chor Bazaar to the bustle of Crawford Market, the energy of Mumbai's traditional markets is a key part of the city's charm. And while these markets will always have their place, the city also boasts a thriving e-commerce scene, ranking among the largest in the region. Driven by online sales in everything from snacks to licensed sports merchandise to children's apparel, the local industry is worth billions, with companies actively recruiting to meet the demands of continued growth.
