Data Engineer

• Minimum Relevant Experience: 5 Years
• Start Date: Immediate
• Key Skills: Informatica, AWS Cloud Services, Databricks, Python, Oracle SQL, Tableau

Position Summary / Project Description:

A ETL Data Engineer will perform ETL development in Informatica IDMC and Databricks. This role involves the ETL developmental efforts, ensuring seamless transition with common tools, and leveraging the advanced features of IDMC and Databricks. The ideal person will have extensive experience in data integration, ETL and hands-on expertise with both IDMC and Databricks.

Role and Responsibilities:

• Developer for data-warehousing, ETL and BI projects in Informatica IDMC and Databricks.
• Responsible for all technical development work and support the leads on technical design and systems impact analysis.
• Hands on experience in Informatica IDMC for creating mappings, workflows, setting up Secure Agents etc.
• Integrate data from various sources, both internal and external, into AWS and Databricks environments, ensuring data consistency and quality, while leveraging Informatica IDMC for data integration, transformation, and governance.
• Develop ETL (Extract, Transform, Load) processes to cleanse, transform, and enrich data, making it suitable for analytical purposes using Databricks' Spark capabilities and Informatica IDMC for data transformation and quality.
• Monitor and optimize data processing and query performance in both AWS and Databricks environments, making necessary adjustments to meet performance and scalability requirements. Utilize Informatica IDMC for optimizing data workflows.
• Implement security best practices and data encryption methods to protect sensitive data in both AWS and Databricks, while ensuring compliance with data privacy regulations. Employ Informatica IDMC for data governance and compliance.
• Maintain clear and comprehensive documentation of data infrastructure, pipelines, and configurations in both AWS and Databricks environments, with metadata management facilitated by Informatica IDMC.
• Collaborate with cross-functional teams, including data scientists, analysts, and software engineers, to understand data requirements and deliver appropriate solutions across AWS, Databricks, and Informatica IDMC.
• Identify and resolve data-related issues and provide support to ensure data availability and integrity in both AWS, Databricks, and Informatica IDMC environments.
• Optimize AWS, Databricks, and Informatica resource usage to control costs while meeting performance and scalability requirements.
• Stay up to date with AWS, Databricks, Informatica IDMC services, and data engineering best practices to recommend and implement new technologies and techniques. Conduct user requirement analysis for the development / implementation of new systems and enhancements to existing systems.
• Evaluate potential solutions and make recommendations to resolve business problems.
• Involvement in the system integration testing phase prior to implementation. Plan and coordinate end-user training for any system implementations or enhancements.
• Liaise closely with business users and build good rapport.
• Liaise closely with vendors in project implementation, application testing, supporting application patches and upgrades in accordance with project methodologies and policies.
• Support the team in defining project requirements, tracking and documentation.

Requirements / Qualifications:

• Degree in Computer Science, Information Technology, Computer Engineering or equivalent
• Minimum 6 years of experience in data engineering, with expertise in AWS or Azure services, Databricks, and/or Informatica IDMC.
• Understanding and hands on experience in all phases of project lifecycle & has implementation involvement in at least 4 project cycles.
• Experience in AWS Services focusing on ETL and data processing using ETL software: Informatica PowerCenter, IDMC, Informatica Data Engineering Integration (DEI), Databricks
• Strong understanding of data integration concepts, ETL process and data quality management.
• Accomplished in designing and implementing ETL data mappings, transformations, and workflows.
• Evaluate potential technical solutions and make recommendations to resolve data issues especially on performance assessment for complex data transformations and long running data processes.
• Strong in Business Intelligence data model design and proficient in BI software such as Oracle Analytics Server (OAS), Tableau.
• Strong knowledge of Oracle SQL and NoSQL databases, equipped with key SQL Skillsets on databases such as Oracle, Teradata and MS-SQL Server.
• Working knowledge on BI standard languages such as Python, C#, Java, VBA.
• Excellent problem-solving and analytical skills.
• Strong communication (with effective written and verbal communications skills) and collaboration skills.
• Effective written and verbal communication skills.
• Team player to ensure team meets delivery targets.
• AWS Associate/AWS Professional/AWS Specialty certification is preferred.
• Databricks certifications, IDMC certifications, Tableau Certifications are a plus.
• Experience in healthcare industry, with broad understanding of healthcare business processes like patient administration, patient medical and non-medical services are a plus.