Build and operate scalable batch and streaming data pipelines, connectors, and RAW-to-CUBE transformations using Python, Spark, Databricks, Kafka, dbt, SQL, and Delta Lake. Ensure data quality, schema validation, observability, performance, cost efficiency, and reliable recovery. Develop governed data products and feature-serving interfaces for machine learning models, agents, and downstream applications.
Job Summary
CONVO is seeking a Senior Data Engineer to build reliable data pipelines and connectors powering its CPG-focused agentic platform. You’ll develop scalable batch and streaming pipelines using Python, Spark, Databricks, Kafka, dbt, and Delta Lake, transforming source data into governed RAW and CUBE layers while ensuring data quality, performance, observability, and reliability. You’ll also build data-product and feature-serving interfaces that enable ML models, agents, and downstream applications to consume trusted data efficiently.
Technical mission- Build reliable connectors and RAW-to-CUBE pipelines across batch and streaming paths, and expose governed data and ML features to downstream services.
- Develop Python/Spark ingestion and transformation pipelines from source systems into RAW and Enriched CUBE layers.
- Implement streaming paths alongside scheduled workloads.
- Create dbt/SQL transformations, data-quality controls and contract-validation checks.
- Optimize Delta Lake layout, SQL performance, partitioning and incremental processing.
- Build feature-serving and data-product interfaces for models, agents and applications.
- Instrument pipelines for lineage, freshness, throughput, failure recovery and cost.
- Minimum experience: 5+ years in data engineering, including 3+ years delivering production Spark or python pipelines.
- Advanced Python and production Apache Spark/Databricks development.
- Kafka or comparable event-streaming technology.
- Strong SQL performance tuning and dimensional/data-product implementation.
- dbt and Delta Lake, including incremental patterns and table optimization.
- Batch and streaming reliability patterns: idempotency, checkpointing, replay and late-arriving data.
- Automated data quality, observability and schema-contract testing.
- ML feature stores or online/offline feature consistency.
- CPG, retail, ERP, POS or syndicated-data pipelines.
- Kubernetes-based data workloads and cloud cost optimization.
- Production connectors and RAW-to-CUBE pipelines with automated tests.
- Batch/streaming operational dashboards and recovery procedures.
- Documented data products and feature-serving interfaces.
- Performance and cost baselines for the implemented workloads.
- Works under the Data Architect with source-system owners, ML/optimization teams, platform infrastructure and downstream application teams.
Similar Jobs
Artificial Intelligence • Fintech • Machine Learning • Software • App development • Conversational AI • Generative AI
Migrate SQL Server data products and analytical workloads to a modern cloud platform. Redesign legacy SQL, ETL processes, and business logic; build scalable pipelines and analytical models; validate data quality; collaborate with stakeholders; and improve platform reliability, performance, cost efficiency, and AI-assisted data engineering automation.
Top Skills:
BigQueryCi/CdCloud ComposerETLGCPGoogle Cloud StorageIamMs Sql ServerPythonSQL
Agency • Information Technology
Build and maintain Snowflake and SQL Server data pipelines for an operational data store, including ingestion, transformation, schema modeling, migrations, CI/CD, data quality testing, and legacy batch-to-API modernization. Collaborate with domain teams and architects to support customer, loan, payment, and interaction data. Ensure data accuracy, lineage, observability, and performance while applying generative AI tools to accelerate development and testing.
Top Skills:
AlationApache AirflowCollibraDagsterDbtFlywayGenerative AiGitGithub ActionsGitlab CiGreat ExpectationsInformaticaPostgresPrefectPythonRest ApisSchemachangeSnowflakeSnowflake Dynamic TablesSnowflake TasksSQLSQL ServerTerraform
Blockchain
Design, deploy, and maintain AWS and Databricks data infrastructure, data lakes, and scalable ETL pipelines. Build custom ingestion services and connectors using Airbyte, Go, and Python; optimize performance, reliability, monitoring, and data quality. Implement cloud data governance, security, access controls, encryption, and compliance practices while collaborating with platform engineers, analysts, and stakeholders.
Top Skills:
AirbyteAmazon RedshiftAmazon S3AWSAws GlueAws IamAws LambdaDatabricksGoPythonSQL
What you need to know about the Mumbai Tech Scene
From haggling for the best price at Chor Bazaar to the bustle of Crawford Market, the energy of Mumbai's traditional markets is a key part of the city's charm. And while these markets will always have their place, the city also boasts a thriving e-commerce scene, ranking among the largest in the region. Driven by online sales in everything from snacks to licensed sports merchandise to children's apparel, the local industry is worth billions, with companies actively recruiting to meet the demands of continued growth.
