Big Data Platform Engineer

Role description

·      You are operating GlobalData Platform components (VM Servers, Kubernetes, Kafka) and applications(Apache stack, Collibra, Dataiku and similar)

·      Implement automation ofinfrastructure, security components, and Continuous Integration &Continuous Delivery for optimal execution of data pipelines (ELT/ETL).

·      Develop solutions tobuild resiliency in data pipelines with platform health checks, monitoring, andalerting mechanisms, quality, timeliness, recency, and accuracy of datadelivery are improved

·      Apply DevSecOps &Agile approaches to deliver the holistic and integrated solution in iterativeincrements.

·      Liaison and collaboratewith enterprise security, digital engineering, and cloud operations to gainconsensus on architecture solution frameworks.

·      Review system issues,incidents, and alerts to identify root causes and continuously implementfeatures to improve platform performance.

·      Be current on the latestindustry developments and technology trends to effectively lead and design newfeatures/capabilities.

Experience

·      You have 5+ years ofexperience in building or designing large-scale, fault-tolerant, distributedsystems

·      Migration experience ofstorage technologies (e.g. HDFS to S3 Object Storage)

·      Integration of streamingand file-based data ingestion /consumption (Kafka, Control M, AWA)

·      Experience in DevOps,data pipeline development, and automation using Jenkins and Octopus (optional:Ansible, Chef, XL Release, and XL Deploy)

·      Experience predominatelywith on-prem Big Data architecture, cloud migration experience might come handy

·      Hands-on experience inintegrating Data Science Workbench platforms (e.g. Dataiku)

·      Experience of agileproject management and methods (e.g., Scrum, SAFe)

·      Supporting allanalytical value streams from enterprise reporting (e.g. Tableau) to datascience (incl. ML Ops)

Skills

·      Hands-on workingknowledge of large data solutions (for example: data lakes, delta lakes, datameshes, data lakehouses, data platforms, data streaming solutions...)

·      In-depth knowledge andexperience in one or more large scale distributed technologies including butnot limited to: Hadoop ecosystem, Kafka, Kubernetes, Spark

·      Expert in Python andJava or another static language like Scala/R, Linux/Unix scripting, Jinjatemplates, puppet scripts, firewall config rules setup

·      VM setup and scaling(pods), K8S scaling, managing Docker with Harbor, pushing Images through CI/CD

·      Experience using dataformats such as Apache Parquet, ORC or Avro Experience in machine learningalgorithms is a plus.

·      Good knowledge of Germanis beneficial, excellent command of English is essential

·      Knowledge of financialsector and its products

·      Higher education (e.g."Fachhochschule", "Wirtschaftsinformatik")