Azure Engineer

Key Responsibilities:

Design and deploy Azure data platform solutions including Azure Synapse Analytics, Azure Data Factory, Azure Data Lake Storage (ADLS Gen2), and Databricks/Apache Spark environments.

Build and maintain data ingestion pipelines using Azure Data Factory and Azure

Synapse pipelines to support batch and real-time data processing. Implement and manage Azure Databricks workspaces and Apache Spark clusters for large-scale data processing and analytics workloads.

Configure and optimize Azure Synapse Analytics dedicated SQL pools, serverless SQL pools, and Spark pools for data warehousing and analytics. Deploy and manage Azure Machine Learning workspaces, compute instances, and ML pipelines for model training and deployment.

Implement Azure OpenAI service integrations for generative AI and LLM-based applications within the data platform. Design and implement data lake architecture following medallion architecture (bronze, silver, gold layers) for data organization and governance.

Monitor data platform resources using Azure Monitor, Log Analytics, and service-specific monitoring tools to ensure optimal performance and availability. Manage Azure Active Directory, role-based access control (RBAC), and data security policies including encryption, network security, and private endpoints.

Troubleshoot and resolve L2 escalations related to data pipeline failures, Spark job performance issues, and data platform infrastructure. Implement backup, disaster recovery, and business continuity solutions for critical data assets.

Build and maintain Azure DevOps pipelines for automated deployment of Terraform configurations across development, staging, and production environments.

Required Qualifications:

4+ years of experience working with Microsoft Azure cloud services, with focus on data and analytics platforms.

Strong hands-on experience with Azure Synapse Analytics, Azure Data Factory, and Azure Data Lake Storage.

Proficiency with Apache Spark and distributed data processing concepts. Experience deploying and managing Azure Databricks environments.

Knowledge of Azure Machine Learning services and ML lifecycle management.

Familiarity with Azure OpenAI service and integration patterns.

Understanding of data warehousing concepts, ETL/ELT processes, and data modeling. Experience with data lake architecture patterns and best practices.

Proficiency in PowerShell, Azure CLI, Python, or PySpark for automation and data processing. Understanding of networking concepts including private endpoints, VNets, NSGs, and service endpoints for secure data platform deployments.

Experience with Infrastructure as Code tools like ARM templates, Bicep, or Terraform.

Microsoft Azure certifications such as AZ-104 (Azure Administrator), DP-203 (Azure Data Engineer), or AZ-305 (Azure Solutions Architect) preferred.

