At 3Pillar Global, we specialize in building impactful software products that power digital businesses. We are looking for a talented Software Engineer who will not only write code but also deliver exceptional product capabilities for our clients. Your contributions will play a pivotal role in achieving our passion for client success, solidifying our position as a leading product development services company.
We are hiring a Senior Data Engineer coming with a strong experience in building data pipeline using PySpark, Python, ETL and Hadoop. Should have experience in Shell scripting and Banking domain.
The incumbent has to work from Office location - Vikroli, Mumbai for all weekdays.
Responsibilities:
- Design, develop, and maintain data pipelines and workflows using Hadoop and PySpark to process large volumes of data efficiently.
- Implement data ingestion, transformation, and storage solutions to support business requirements.
- Develop and optimize Python scripts and algorithms for data manipulation, analysis, and modeling tasks.
- Design and manage database systems, including schema design, indexing, and performance optimization.
- Collaborate with cross-functional teams to understand data requirements and deliver solutions that meet business needs.
- Perform data quality assessment, validation, and cleansing to ensure accuracy and integrity of data.
- Monitor and troubleshoot data pipeline issues, performance bottlenecks, and system failures.
- Stay up to date with emerging technologies and best practices in data engineering and analytics.
Qualification:
- Bachelor's degree in Computer Science, Engineering, or related field.
- Should have a minimum of 5 years of experience in the field of Database or Data Analytics
- Minimum of 3 years of experience in data engineering, with expertise in Hadoop, PySpark, Python, and database technologies.
- Strong proficiency in Hadoop ecosystem components such as HDFS, MapReduce, Hive, and Spark.
- Experience with PySpark for data processing and analysis tasks.
- Proficiency in Python programming for scripting and automation.
- Solid understanding of database concepts and hands-on experience with SQL and NoSQL databases.
- Experience with data modeling, schema design, and ETL processes.
- Strong problem-solving skills and ability to work independently and collaboratively in a fast-paced environment.
- Excellent communication and interpersonal skills, with the ability to effectively communicate technical concepts to non-technical stakeholders.
Top Skills
What We Do
3Pillar Global builds breakthrough software products that power digital businesses. 3Pillar is an innovative product development partner whose solutions drive rapid revenue, market share, and customer growth for industry leaders in Software and SaaS, Media and Publishing, Information Services, and Retail. Leveraging a lean and agile approach, 3Pillar delivers value-generating, digital solutions with specialized product strategy and management, user experience design, as well as software and data engineering expertise across mobile, cloud, and disruptive technologies.
