Big Data Cloud Engineer (Banking Sector, HDFS, YARN, Kafka, Linux, AWS, Shell scripting)

Responsibilities:

• Administer, maintain, and support large-scale Hadoop and Cloudera (CDH/CDP) environments across enterprise distributed infrastructures.
• Design and manage Hadoop cluster architecture, deployment, configuration, and lifecycle management for highly available production environments.
• Manage and optimize core Big Data ecosystem components including HDFS, YARN, MapReduce, Spark, PySpark, Kafka, Hive, Impala, HBase, Oozie, and Zookeeper.
• Perform administration, monitoring, troubleshooting, and performance tuning of Hadoop clusters to ensure high availability, scalability, and operational stability.
• Support and maintain real-time and batch data processing platforms using Spark, Kafka Streams, Hive, and HBase technologies.
• Configure and support Hadoop ecosystem tools including Hue, Arcadia, Airflow, NiFi, Sqoop, Datameer, Splunk, and AEN.
• Implement and manage enterprise-grade security frameworks using Kerberos, Ranger, Sentry, SSL, IAM policies, and access control mechanisms.
• Manage and support AWS cloud infrastructure, including EC2, S3, EMR, IAM, and VPC services for Big Data workloads.
• Perform capacity planning, cluster expansion, metadata management, partition optimization, and resource tuning to improve platform performance.
• Develop and maintain automation utilities using Shell scripting, Python, PySpark, and Linux scripting for operational efficiency and proactive monitoring.
• Execute OS patching, cluster upgrades, maintenance activities, backup strategies, and infrastructure management with minimal business disruption.
• Monitor production environments using enterprise monitoring and observability tools including NewRelic and other operational monitoring platforms.
• Lead incident management, root cause analysis (RCA), problem resolution, and production support activities for mission-critical Big Data platforms.
• Collaborate with DevOps, infrastructure, database, and application teams to support enterprise analytics and data engineering initiatives.
• Participate in Agile delivery processes, operational reviews, and continuous improvement initiatives.
• Mentor junior team members and contribute to technical knowledge sharing and operational best practices.
• Support enterprise platforms and applications within Financial/Banking domain environments with strict compliance and availability requirements.
• Provide support for PeopleSoft Administration and related enterprise integration environments where required.

Requirements:

• 10+ years of experience in Hadoop Administration and Cloudera Administration (CDH/CDP).
• Experience in Financial/Banking Sector
• Strong experience in Hadoop Cluster Architecture, Deployment, Administration, and Performance Tuning.
• Hands-on experience in HDFS, YARN, Kafka, Spark, Hive, Impala, MapReduce, HBase, Oozie, and Zookeeper.
• Strong experience in Linux Administration and enterprise infrastructure troubleshooting.
• Hands-on experience in AWS cloud services, including EC2, S3, EMR, IAM, and VPC.
• Experience in Shell scripting, Python, and PySpark for automation and operational support.
• Experience in PeopleSoft Administration and Oracle GoldenGate.
• Strong understanding of data security, authentication, authorization, and compliance frameworks including Kerberos, Ranger, and Sentry.
• Experience in monitoring, incident management, production support, and performance analysis using tools such as NewRelic.
• Experience in supporting large-scale, high-availability, multi-region production environments.
• Experience working with CI/CD pipelines, DevOps practices, and GitHub/Jenkins-based automation frameworks.
• Experience in Hadoop ecosystem tools including Hue, Arcadia, Airflow, NiFi, Sqoop, Datameer, Splunk, and AEN.
• Experience in PeopleSoft Administration and enterprise application support environments.
• Experience in Financial/Banking sector projects and mission-critical enterprise systems is highly preferred.
• Strong analytical, troubleshooting, stakeholder management, and communication skills.