Data Engineer (Databricks)

• 1-year contract, renewable
• Government project
• Hybrid work arrangement

The Data Engineer will perform a crucial role in designing, building, and maintaining robust infrastructure and systems across digital products. You will work under the guidance of senior team members to support the development of data pipelines and infrastructure that ensure clean, accurate, and timely data is available for business analytics, decision-making, and AI applications.

Responsibility:

Data Infrastructure Support

• Assist in designing and developing data pipelines and architectures to collect, process, harmonize and store data from various source systems across digital products
• Support the building of data pipelines that integrate data from multiple platforms and services
• Help maintain data lakes and database infrastructure to support analytical, reporting, and AI workloads across the ecosystem

Data Quality and Monitoring

• Implement data validation, cleansing, and normalisation processes under supervision to ensure data quality and integrity
• Monitor data pipelines and systems to identify potential issues and performance bottlenecks
• Assist in maintaining data governance frameworks that support compliance and data security requirements

System Maintenance and Documentation

• Support the maintenance of data infrastructure supporting digital products
• Assist in troubleshooting and resolving data-related issues under guidance from senior team members
• Create and maintain documentation of data engineering processes and system configurations

Stakeholder collaboration

• Work closely with senior data engineers, product teams, analysts, and stakeholders to understand business data requirements and translate them into technical specifications and scalable solutions
• Design, develop and deploy data tables, visualisation and marts for business reporting and other purposes

Professional Development

• Stay current with emerging technologies and trends in data engineering through training and mentorship
• Participate in continuous learning opportunities to develop technical skills and domain knowledge
• Contribute to process improvements and suggest enhancements to existing data systems
• Participate in knowledge sharing sessions and contribute to team learning initiatives

Requirement

• A Bachelors Degree, in Computer Science, Data Science, Engineering, Information Technology, or related technical field.
• 1-3 years of experience in data engineering, software development, or related technical roles
• Experience with any cloud platform (AWS, Azure, or GCP)
• Knowledge and experience business intelligence tools such as Power BI, AWS QuickSight, or similar platforms
• Experience with data visualisation tools and techniques
• Familiarity with agile development methodologies
• Previous project experience involving data processing or analytics
• Understanding of data governance policies, access controls, and security best practices in government environments.
• Possess any of the certifications would be advantageous: AWS Certified Data Engineer, Databricks Certified Data Engineer

Technical Skills

• Proficiency in at least one programming language such as Python, Java, or Scala for data processing and scripting
• Basic knowledge of database systems, both relational and non-relational, and their query languages (SQL knowledge essential)
• Understanding of data modelling and schema design principles
• Familiarity with data integration and ETL (Extract, Transform, Load) concepts and processes
• Basic experience with version control systems such as Git

Cloud and Platform Knowledge

• Basic familiarity with cloud platforms, preferably AWS services such as S3, EC2, and RDS
• Exposure to big data technologies and frameworks such as Spark or similar platforms
• Experience with Databricks and implementing batch/real-time data pipeline