Senior Data Engineer – Databricks

         Job Description

·      We are looking for an experienced Senior Data Engineer – Databricks to design, develop, and maintain scalable data pipelines on the Databricks platform. 

·      The role requires strong expertise in PySpark, Databricks, and modern data engineering practices, along with experience modernizing legacy data pipelines and building high-performance data solutions in cloud environments. 

Key Responsibilities 

·      Design, build, and maintain scalable data pipelines using Databricks and PySpark. 

·      Develop end-to-end data workflows including data ingestion, transformation, and consumption. 

·      Optimize Spark jobs, cluster configurations, and pipeline performance. 

·      Manage and orchestrate workflows using Databricks Jobs and notebooks. 

·      Refactor legacy ETL pipelines into modern PySpark-based ELT frameworks. 

·      Implement data quality checks, monitoring, and error handling mechanisms. 

·      Design and maintain Delta Lake tables with proper optimization strategies. 

·      Collaborate with data architects, analysts, and infrastructure teams to deliver reliable data solutions. 

·      Troubleshoot and resolve production data pipeline issues.

Required Skills & Qualification

·      Strong Data Engineering fundamentals including ETL/ELT pipeline design. 

·      Hands-on experience with PySpark (DataFrames API, Spark SQL, performance tuning). 

·      Experience with Databricks platform including workspace, clusters, notebooks, and job orchestration. 

·      Knowledge of Delta Lake features such as ACID transactions and schema evolution. 

·      Strong Python programming for data processing and automation. 

·      Experience with data modelling (Dimensional modeling, Data Vault, or Lakehouse architecture). 

·      Strong SQL skills for data transformation and analysis. 

·      Experience with cloud platforms (Azure, AWS, or GCP). 

·      Experience with Git version control and CI/CD practices. 

·      Minimum 8 years of experience in data engineering or related roles. 

·      2–3 years of hands-on experience with Databricks. 

·      Experience building and maintaining production-grade data pipelines at scale. 

·      Proven experience modernizing legacy data pipelines. 

·      Experience working in Agile development environments. 

·      Certifications (Mandatory) Databricks Certified Data Engineer Associate OR Databricks Certified Data Engineer Professional