Data Engineer

Job Title: Data Engineer

Key Responsibilities:

-         Develop, construct, test, and maintain scalable data systems, pipelines, and architectures (data lakes, warehouses).

-         Collect raw data from various sources, clean it, and transform it into usable formats (ETL/ELT).

-         Work with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions.

-         Improve existing data frameworks, monitor workflows, and troubleshoot issues.

-         Implement data governance, validation, and security measures.

Qualifications & Skills:

Education: Polytechnic Diploma or bachelor’s degree in computer science, Data Analytics, Business Intelligence or any related field (or equivalent experience).  

Experience: 

-         Programming: Strong Python and SQL skills are fundamental, with Java/Scala useful for big data.

-         Databases: Experience with relational (PostgreSQL, MySQL) and NoSQL databases (MongoDB).

-         Cloud Platforms: Proficiency in major cloud providers (AWS, Azure, GCP) for data services (S3, Redshift, EMR, etc.).

-         Big Data: Working knowledge of tools like Apache Spark, Kafka, and Hadoop.

-         ETL/ELT: Designing and implementing data extraction, transformation, and loading processes.

-         Data Warehousing: Concepts and tools for storing large datasets (e.g., Redshift, Snowflake).

-         Orchestration: Tools like Apache Airflow for scheduling and managing workflows.

-         Data Modeling: Designing efficient database and data warehouse schemas.

Technical Skills: 

-         Programming: Python, Java, SQL (essential).

-         Databases: Relational (SQL) & NoSQL databases.

-         Big Data Tools: Hadoop, Spark (often).

-         Cloud Platforms: AWS, Azure, GCP (increasingly common).

-         ETL/ELT Tools: Expertise in data integration tools.

-         Software Engineering: Strong understanding of data structures and algorithms.

Soft Skills:  

-         Strong analytical and problem-solving abilities.  

-         Good communication skills to collaborate with non-technical teams.  

-         Eagerness to learn and adapt in a fast-paced environment

-         Open to learn new software or BI tools in the market.

Nice to Have: 

-         Experience with BI or Reporting tools (publishing, scheduling, permissions)

- Knowledge of GitHub, GitLab

-         Knowledge of SDLC & DDLC

-         Understanding of Agile/Scrum methodologies.