Senior Data Engineer (ETL, Hive, Cloudera/Spark)

Job Description

Development & Tuning

• Apache Spark (Scala, Python)
• Spark performance tuning
• Hive query and table optimization
• HBase data modeling and tuning

Cloudera Platform Components

• HIVE
• HBASE
• Phoenix
• YARN
• HDFS
• Data Engineering
• ETL job design and development
• Data pipelines and batch processing

Application & Infrastructure

• JBoss
• MariaDB
• Linux environments

Governance & Tooling

• JIRA
• Change Request (CRQ) processes

Scope of Work

•Perform application-aware performance analysis for AML data pipelines.

•Analyze and optimize pipelines running on Cloudera, Spark, Hive, HBase, JBoss, and MariaDB.

•Tune Spark, Hive, and HBase jobs, queries, and tables for performance and scalability.

•Review ETL job design and data pipeline architecture for efficiency, resilience, and scalability.

•Identify misconfigurations or misuse causing performance degradation or recurring issues.

•Assess and recommend tuning for Cloudera, Phoenix, YARN, and related platform configurations.

•Support large-scale data ingestion, transformation, and downstream data delivery.

•Participate in joint Root Cause Analysis (RCA) with internal and vendor teams.

•Support performance benchmarking and validation of tuning recommendations.

•Support application and system integrations with Cloudera and AML platforms.

•Build and refine data validation dashboards and data growth monitoring tools.

•Troubleshoot slow-running jobs, data skews, shuffle issues, GC challenges, YARN resource pressure, and cluster constraints.

• Improve performance of API/UI components interacting with Big Data pipelines.