Design, develop, and optimize high-performance data pipelines using Apache Spark, ensuring reliability and scalability. Collaborate with teams to implement best practices for data quality and security.
Mactores is a trusted leader among businesses in providing modern data platform solutions. Since 2008, Mactores have been enabling businesses to accelerate their value through automation by providing End-to-End Data Solutions that are automated, agile, and secure. We collaborate with customers to strategize, navigate, and accelerate an ideal path forward with a digital transformation via assessments, migration, or modernization.
We are seeking a highly skilled and innovative Spark Engineer to join our team. In this role, you will design, develop, optimize, and operationalize high-performance data pipelines and applications using Apache Spark. This role requires hands-on expertise in distributed data processing, ETL engineering, performance tuning, cluster management, and working with cross-functional teams to deliver reliable, scalable, and efficient data solutions
What will you do
- Architect, design, and build scalable data pipelines and distributed applications using Apache Spark (Spark SQL, DataFrames, RDDs)
- Develop and manage ETL/ELT pipelines to process structured and unstructured data at scale.
- Write high-performance code in Scala or PySpark for distributed data processing workloads.
- Optimize Spark jobs by tuning shuffle, caching, partitioning, memory, executor cores, and cluster resource allocation.
- Monitor and troubleshoot Spark job failures, cluster performance, bottlenecks, and degraded workloads.
- Debug production issues using logs, metrics, and execution plans to maintain SLA-driven pipeline reliability.
- Deploy and manage Spark applications on on-prem or cloud platforms (AWS, Azure, or GCP).
- Collaborate with data scientists, analysts, and engineers to design data models and enable self-serve analytics.
- Implement best practices around data quality, data reliability, security, and observability.
- Support cluster provisioning, configuration, and workload optimization on platforms like Kubernetes, YARN, or EMR/Databricks.
- Maintain version-controlled codebases, CI/CD pipelines, and deployment automation.
- Document architecture, data flows, pipelines, and runbooks for operational excellence
What we are looking for
- Bachelor’s degree in Computer Science, Engineering, or a related field.
- 4+ years of experience building distributed data processing pipelines, with deep expertise in Apache Spark.
- Strong understanding of Spark internals (Catalyst optimizer, DAG scheduling, shuffle, partitioning, caching).
- Proficiency in Scala and/or PySpark with strong software engineering fundamentals.
- Solid expertise in ETL/ELT, distributed computing, and large-scale data processing.
- Experience with cluster and job orchestration frameworks.
- Strong ability to identify and resolve performance bottlenecks and production issues.
- Familiarity with data security, governance, and data quality frameworks.
- Excellent communication and collaboration skills to work with distributed engineering teams.
- Ability to work independently and deliver scalable solutions in a fast-paced environment
You will be preferred if
- Experience with Databricks, AWS EMR, Glue Spark, or GCP Dataproc.
- Familiarity with workflow orchestration tools like Apache Airflow, Dagster, or Prefect.
- Exposure to streaming platforms such as Kafka, Kinesis, or Pub/Sub.
- Experience running Spark workloads on Kubernetes.
- Familiarity with data warehouse ecosystems (Snowflake, BigQuery, Redshift, Iceberg, Delta Lake, Hudi).
- Understanding of DevOps practices, CI/CD, and IaC (Terraform, CloudFormation).
- Knowledge of distributed logging and monitoring tools (Grafana, Prometheus, CloudWatch, ELK).
- Prior experience in high-scale production environments or data platform teams
Life at Mactores
We care about creating a culture that makes a real difference in the lives of every Mactorian. Our 10 Core Leadership Principles that honor Decision-making, Leadership, Collaboration, and Curiosity drive how we work.
1. Be one step ahead
2. Deliver the best
3. Be bold
4. Pay attention to the detail
5. Enjoy the challenge
6. Be curious and take action
7. Take leadership
8. Own it
9. Deliver value
10. Be collaborative
We would like you to read more details about the work culture on https://mactores.com/careers
The Path to Joining the Mactores Team
At Mactores, our recruitment process is structured around three distinct stages:
Pre-Employment Assessment:
You will be invited to participate in a series of pre-employment evaluations to assess your technical proficiency and suitability for the role.
Managerial Interview: The hiring manager will engage with you in multiple discussions, lasting anywhere from 30 minutes to an hour, to assess your technical skills, hands-on experience, leadership potential, and communication abilities.
HR Discussion: During this 30-minute session, you'll have the opportunity to discuss the offer and next steps with a member of the HR team.
At Mactores, we are committed to providing equal opportunities in all of our employment practices, and we do not discriminate based on race, religion, gender, national origin, age, disability, marital status, military status, genetic information, or any other category protected by federal, state, and local laws. This policy extends to all aspects of the employment relationship, including recruitment, compensation, promotions, transfers, disciplinary action, layoff, training, and social and recreational programs. All employment decisions will be made in compliance with these principles.
Note: Please answer as many questions as possible with this application to accelerate the hiring process.
Top Skills
Apache Airflow
Spark
AWS
Azure
CloudFormation
Cloudwatch
Databricks
Databricks
Elk
Emr
GCP
Grafana
Kafka
Kinesis
Kubernetes
Prometheus
Pyspark
Scala
Terraform
Yarn
Similar Jobs
Artificial Intelligence • Cloud • Information Technology • Sales • Security • Software • Cybersecurity
The Systems Administrator will manage cloud infrastructure and enterprise endpoints, ensuring system stability and security. Responsibilities include supporting cloud operations, endpoint management, and incident response alongside cross-regional collaboration.
Top Skills:
Active DirectoryAutomoxAWSAzureBashChrome OsEntra IdGCPGoogle Admin ConsoleGoogle WorkspaceJAMFmacOSMicrosoft IntuneOktaPowershellPythonSlackWindowsWorkspaceoneZoom
Artificial Intelligence • Consumer Web • Edtech • HR Tech • Information Technology • Software • Conversational AI
Negotiate and manage customer contracts, draft legal documents, ensure compliance, and collaborate with internal and external teams on contract terms.
Top Skills:
PeoplesoftSalesforceWord
Artificial Intelligence • Enterprise Web • Information Technology • Productivity • Sales • Software • Database
As a Senior Backend Engineer at Apollo.io, you'll design scalable backend solutions, mentor teammates, and work cross-functionally to enhance product quality and performance.
Top Skills:
AIAnsibleDockerElasticsearchKubernetesMongoDBNode.jsReactRedisReduxRubyRuby On RailsTerraform
What you need to know about the Mumbai Tech Scene
From haggling for the best price at Chor Bazaar to the bustle of Crawford Market, the energy of Mumbai's traditional markets is a key part of the city's charm. And while these markets will always have their place, the city also boasts a thriving e-commerce scene, ranking among the largest in the region. Driven by online sales in everything from snacks to licensed sports merchandise to children's apparel, the local industry is worth billions, with companies actively recruiting to meet the demands of continued growth.
