Infrastructure Data Engineer

Job Title

Job Description Summary

Job Description

We are seeking a highly skilled Azure Cloud Engineer with deep expertise in designing, implementing, and optimizing Azure Databricks environments. This role will also support a broad range of Azure services—including Web Applications, Azure Data Factory, Application Gateway, Storage Accounts, Lakebase, Azure SQL/Databases, and Azure Monitor—to ensure our cloud ecosystem is secure, scalable, cost-efficient, and aligned with the Five Pillars of the Azure Well-Architected Framework: Operational Excellence, Security, Reliability, Performance Efficiency, and Cost Optimization.

The ideal candidate combines strong hands-on engineering skills with a strategic mindset for cloud governance, automation, and continuous improvement.

Key Responsibilities

Primary Focus: Azure Databricks Engineering

• ·Design, deploy, and optimize Azure Databricks workspaces, clusters, jobs, and pipelines.
• Implement best practices for cluster sizing, autoscaling, job orchestration, and cost governance.
• Collaborate with data engineering teams to build scalable ETL/ELT pipelines using Databricks notebooks, Delta Lake, and Spark.
• Integrate Databricks with Azure Data Lake, Azure SQL, Azure Data Factory, and other platform services.
• Implement security controls including Unity Catalog, RBAC, service principals, secret scopes, and network isolation.
• Monitor Databricks performance and reliability using Azure Monitor, Log Analytics, and Databricks native tools.

Azure Platform Engineering Responsibilities

Azure Application & Integration Services

• Deploy, configure, and maintain Azure Web Apps, Function Apps, and API Management.
• Build and maintain Azure Data Factory pipelines for data ingestion, transformation, and orchestration.
• Configure and optimize Azure Application Gateway, WAF policies, routing rules, and SSL/TLS certificates.

Azure Storage & Database Services

• Manage Azure Storage Accounts, lifecycle policies, replication, encryption, and access controls.
• Support Azure SQL, Cosmos DB, and other database services including performance tuning and security hardening.

Monitoring, Governance & Optimization

• Use Azure Monitor, Log Analytics, Application Insights, and Azure Advisor to proactively identify and resolve issues.
• Implement alerting, dashboards, and automated remediation workflows.
• Continuously optimize cloud resources for cost, performance, and reliability.
• Contribute to cloud governance standards including tagging, resource organization, RBAC, and policy enforcement.

Automation & Infrastructure as Code

• Build and maintain IaC templates using Terraform, Bicep, or ARM.
• Automate deployments, configuration, and operational tasks using Azure DevOps, GitHub Actions, or PowerShell.
• Support CI/CD pipelines for application and data workloads.

Responsibilities Aligned to the 5 Pillars of the Cloud Operating Model

Operational Excellence

• Develop repeatable deployment patterns and IaC modules.
• Implement monitoring, logging, and alerting standards across all Azure services.
• Document architectures, runbooks, and operational procedures.

Security

• Enforce Zero Trust principles across identity, network, and data layers.
• Implement RBAC, managed identities, Key Vault, private endpoints, and network security groups.
• Support compliance initiatives and security assessments.

Reliability

• Design resilient architectures using availability zones, redundancy, and failover strategies.
• Implement backup, disaster recovery, and business continuity plans.
• Monitor system health and proactively address reliability risks.

Performance Efficiency

• Optimize Databricks clusters, SQL databases, and application services for throughput and latency.
• Conduct performance testing and capacity planning.
• Use Azure Monitor and Advisor insights to drive continuous improvement.

Cost Optimization

• Implement cost-efficient Databricks cluster policies and autoscaling.
• Monitor and optimize Azure consumption across compute, storage, and networking.
• Provide recommendations for reserved instances, rightsizing, and lifecycle management.
   

Required Qualifications

• 3–7+ years of hands-on experience with Microsoft Azure.
• Strong expertise in Azure Databricks, Spark, Delta Lake, and data engineering concepts.
• Experience with Azure services such as Web Apps, Data Factory, Application Gateway, Storage Accounts, and Azure SQL.
• Proficiency with Terraform, Bicep, ARM, or similar IaC tools.
• Strong scripting skills (PowerShell, Python, Bash).
• Experience with Azure Monitor, Log Analytics, and Azure Advisor.
• Solid understanding of cloud networking (VNETs, private endpoints, firewalls, routing).
• Familiarity with the Azure Well-Architected Framework.

Preferred Qualifications

• Azure certifications (AZ-104, AZ-204, DP-203, AZ-305, or Databricks certifications).
• Experience with CI/CD pipelines in Azure DevOps or GitHub Actions.
• Experience with Unity Catalog, Databricks SQL, or MLflow.
• Background in data engineering, DevOps, or cloud architecture.

Soft Skills

• Strong communication and documentation abilities.
• Ability to collaborate across data, application, and infrastructure teams.
• Problem-solving mindset with a focus on automation and continuous improvement.
• Comfortable working in fast-paced, cloud-first environments