Build and optimize scalable data ingestion, transformation, and consumption pipelines on an on-premises lakehouse platform. Develop Spark-based ETL/ELT workloads, Apache Iceberg data models, and Informatica integration workflows. Enable analytics through Denodo, Starburst, and Dremio while implementing data quality, governance, metadata, security, and validation controls. Support batch and near-real-time processing, platform deployment, migration, testing, production rollouts, and performance optimization.
Job Description – Data Engineer
Employment Type: Full-Time
Work Location: KSA
Experience Required: Hands-on experience with Spark development, Informatica integration solutions, and enterprise data platforms.
Years of Experience: 5 to 10
We are looking for a Data Engineer to build, configure, and optimize data ingestion, transformation, and consumption workloads on a modern on-premises lakehouse platform. The ideal candidate will have strong hands-on experience with Spark-based development, enterprise data lake/lakehouse implementations, and Informatica-based integration solutions.
Key Responsibilities- Design and develop scalable data ingestion and transformation pipelines.
- Build, optimize, and maintain Spark-based ETL/ELT workloads.
- Develop and manage data models and datasets using Apache Iceberg tables.
- Implement and support data integration workflows using Informatica.
- Enable analytics and data consumption through Denodo, Starburst, and/or Dremio.
- Develop secure, governed, and scalable data pipelines aligned with enterprise standards.
- Support batch and near real-time data processing requirements.
- Implement data quality, metadata management, and validation controls.
- Collaborate with Platform Engineers on deployment, integration, and operationalization activities.
- Support migration, testing, and production rollout initiatives.
- Troubleshoot and optimize data pipeline performance, reliability, and scalability.
- Apache Spark (PySpark and/or Scala Spark)
- Apache Iceberg
- Informatica Data Integration
- MinIO Object Storage
- Denodo
- SQL and Data Modeling
- Data Warehousing Concepts
- ETL/ELT Development
- Git Version Control
- Starburst and/or Dremio
- Data Governance and Metadata Management
- Data Quality Frameworks
- CI/CD for Data Pipelines
- Python
- Kafka or Streaming Technologies
- Lakehouse Architecture Patterns
- Experience with enterprise data lake and lakehouse implementations
- Performance tuning and optimization of large-scale data workloads
Similar Jobs
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Build and maintain large-scale data platforms, databases, ETL pipelines, and data solutions. Develop optimized SQL, Spark, Kafka, and streaming processes; support data analysts and scientists; perform large-scale extraction and analysis; improve data quality, modeling, architecture, automation, security, and deployment. Collaborate with stakeholders on requirements, investigations, root-cause analysis, and production data artifacts.
Top Skills:
Amazon EmrAmazon S3Apache KafkaAWSClouderaDatabricksDatadogDelta LakeDynatraceIcebergJavaScriptJenkinsOzonePl/SqlPythonRegexRelational DatabasesScalaSparkSpark StreamingSplunkSQLTrino
Fintech • Professional Services • Consulting • Energy • Financial Services • Cybersecurity • Generative AI
Designs, develops, and maintains scalable batch and streaming data pipelines using Python, Scala, Spark, and Databricks. Responsibilities include optimizing large-scale data workloads, building ETL/ELT processes, modeling data warehouses and lakes, implementing data quality and governance standards, troubleshooting production issues, and collaborating with architects, analysts, and data scientists.
Top Skills:
SparkAws DatabricksAzure DatabricksDatabricksDatabricks WorkflowsDataframesDelta LakeEtl/EltGcp DatabricksPythonScalaSpark SqlSQL
Big Data • Food • Hardware • Machine Learning • Retail • Automation • Manufacturing
Design and build scalable cloud-based data solutions, develop ETL pipelines for warehouses and data lakes, implement data quality controls, optimize processing and storage, and collaborate with data teams and stakeholders. The role requires Python, PySpark, SQL, Databricks, AWS services, data modeling, scheduling technologies, and analytics expertise.
Top Skills:
AecorsoftApache AirflowAws Ec2Aws GlueAws LambdaAws RedshiftAws S3DatabricksDatabricks Lakeflow ConnectDatabricks NotebooksDatasphereErwinGeminiGenieKiroLakeflow JobsExcelPower BIPysparkPythonRSQLTableau
What you need to know about the Mumbai Tech Scene
From haggling for the best price at Chor Bazaar to the bustle of Crawford Market, the energy of Mumbai's traditional markets is a key part of the city's charm. And while these markets will always have their place, the city also boasts a thriving e-commerce scene, ranking among the largest in the region. Driven by online sales in everything from snacks to licensed sports merchandise to children's apparel, the local industry is worth billions, with companies actively recruiting to meet the demands of continued growth.
