Senior Data Engineer

Responsibilities

• Develop tested data pipelines to read the data from and load into Hadoop managed tables using yaml files.
• Review the code and also involve in code optimization to resolve performance issues.
• Develop, tested jobs using Control-M scheduler
• work on batch flow for production.
• Prepare shell scripts to do archive data in Hadoop environment.
• Worked on various Hadoop file formats like Parquet, ORC and AVRO files on HDFS using PySpark

Requirements

• Degree in Computer Science
• Have Good understanding of the data model and the entities used in the project, worked with data mappers to resolve mapping issues.
• Minimum 5 years of experience is needed