Key Responsibilities:

Design and develop scalable data pipelines across Hadoop (Hive, Impala, Spark, Kafka, Iceberg) and Teradata environments.
Build ingestion and transformation frameworks using Java, Spark, Python and shell scripts.
Develop full stack applications and internal tools using Python, Shell scripting, and modern web frameworks (e.g., Flask, React).
Create APIs and microservices to expose data and ML models securely to downstream systems and user interfaces.
Collaborate with data scientists to operationalize ML models using Cloudera Machine Learning (CML)
Build and deploy GenAI/LLM-powered applications for intelligent data interaction, summarization, and automation.
Implement enterprise-grade security controls including RBAC, LDAP, Kerberos, Apache Ranger, and row-level access.
Tune and optimize data applications for performance across Hadoop and Teradata, ensuring efficient resource utilization.
Support sandbox environments for prototyping, enabling users to build ML models, dashboards, and data pipelines.

Required Skills & Experience:

Data Engineering: Strong experience with Hadoop ecosystem (Hive, Impala, Spark, Kafka, Iceberg, Ranger, Atlas), Teradata and data pipeline orchestration.
Full Stack Development: Proficiency in Python, Shell scripting, REST APIs, and web frameworks (Flask, React, etc.).
Machine Learning & AI: Hands-on experience with ML platforms (CML), Spark MLlib, Python ML libraries (scikit-learn, XGBoost), and model deployment.
GenAI/LLM Applications: Familiarity with building applications using large language models (e.g., OpenAI, Hugging Face, LangChain) for enterprise use cases.
Security & Governance: Experience with enterprise data security (LDAP, Kerberos, RBAC), data masking, and access control.
Performance Tuning: Proven ability to optimize data applications and queries in large-scale environments (Hadoop, Teradata).
Tools & Platforms: Cloudera Data Platform (CDP), Informatica, QlikSense, Apache Oozie, Git, CI/CD pipelines.
Soft Skills: Strong analytical and problem-solving skills, excellent communication, and ability to work in cross-functional teams.

Data Engineer