Data Engineer (Quantexa / AML)

We are seeking an experienced Data Engineer with excellent expertise in big data platforms, containerized environments, and DevOps practices. In this role, you will design, build, and maintain scalable data pipelines that process large-scale datasets supporting advanced analytics and machine learning solutions.

You will primarily work on the Quantexa platform within the AML domain, ensuring high availability, performance, and reliability of data infrastructure deployed on OpenShift Container Platform (OCP).

Key Responsibilities

Data Engineering & Analytics

• Design, implement, and maintain data transformation, aggregation, and enrichment pipelines to support analytics and machine learning initiatives.
• Process and manage large-scale datasets using Hadoop, Spark, and related technologies.
• Ensure data quality, integrity, and consistency across the entire data lifecycle.
• Implement data governance practices, including data lineage, metadata management, and compliance controls.

Platform & Containerization

• Design and deploy data engineering solutions on OpenShift Container Platform (OCP) using containerization and orchestration techniques.
• Optimize data workflows for containerized deployment and efficient resource utilization.
• Monitor, tune, and enhance data pipeline performance to ensure scalability and reliability.

DevOps & Operations

• Collaborate with DevOps teams to streamline deployments and implement CI/CD pipelines.
• Implement monitoring and logging solutions to ensure health, availability, and performance of the data infrastructure.
• Troubleshoot production issues and perform root cause analysis with a proactive approach.

Collaboration & Leadership

• Work closely with cross-functional teams to translate data requirements into effective engineering solutions.
• Document data pipelines, workflows, and infrastructure configurations for knowledge sharing.
• Stay current with emerging technologies, industry trends, and best practices in data engineering and DevOps.
• Provide technical guidance and mentorship to junior team members, contributing to continuous improvement of analytics capabilities.

Technical & Professional Requirements

Core Experience

• Bachelor’s degree in Computer Science, Information Technology, or a related discipline.
• Minimum 6 years of experience as a Data Engineer in large-scale data environments.
• Hands-on experience developing or maintaining Quantexa solutions (AML domain experience).

Big Data & Processing

• Solid expertise in Hadoop ecosystem tools (HDFS, Hive, Pig, Cloudera).
• Advanced experience with Apache Spark for large-scale data processing.
• Proficiency in SQL and data modeling concepts.

Programming & Scripting

• Proficiency in one or more of the following: Scala, Python, Java.
• Working knowledge of Shell scripting.

Containerization, DevOps & Tooling

• Hands-on experience with OpenShift Container Platform (OCP) and Kubernetes.
• Good understanding of DevOps practices, CI/CD pipelines, and automation tools such as Docker, Jenkins, Ansible, and Bitbucket.
• Experience with enterprise job schedulers such as Control-M.

Monitoring & Observability

• Experience with monitoring and logging tools such as Grafana, Prometheus, and Splunk.

Cloud & Platform Exposure

• Exposure to cloud platforms such as AWS, Azure, or GCP, including data services.
• Quantexa exposure and/or certification.

EA License : 02C3423

EA Personnel : R22108699