Salary Range
SGD 84,000 - SGD 108,000 /year
SGD 7,000 - SGD 9,000/month
Skills Required
Performance TestingPySparkMandatory TestingBig DataData PipelineHadoopData EngineeringPythonData Warehouse ArchitectureData Validation
Job Description
- Experience Requirements:
- Must have QA/Testing Experience.
- Data Testing Experience specifically in Big Data, Hadoop, or Cloud Data Warehouse environments.
- Databricks Experience of testing pipelines within a Databricks environment.
- Automation Focus: Proven track record of moving from manual SQL checks to automated Python-based testing frameworks.
- Mandatory: Databricks Certified Data Engineer Associate (at minimum).
- Preferred: ISTQB Foundation or Advanced Level (Test Automation Engineer).
- Core Technical Skills:
- Data Validation & Frameworks
- Great Expectations / Pandera: Proficiency in using Python-based libraries to define data "contracts" and automated validation suites.
- DLT Expectations: Deep understanding of Delta Live Tables (DLT) expectations (Fail, Drop, Quarantining bad records).
- Advanced SQL: Expert-level SQL for complex data reconciliation, identifying duplicates, and null-value analysis across billions of records.
- Python for QA (PySpark):
- Pytest-Spark: Experience using pytest to write unit tests for PySpark transformations and logic.
- Notebook Testing: Ability to write automated test notebooks that validate Medallion Architecture transitions (Bronze to Silver, Silver to Gold).
- Data Reconciliation: Building Python scripts to perform "source-to-target" counts and checksums across distributed file systems.
- Performance & Integration Testing:
- Scalability Testing: Ability to validate that data pipelines meet performance SLAs when data volume spikes.
- End-to-End Orchestration Testing: Testing the reliability of Databricks Workflows and handling of job failures/retries.
- Schema Evolution: Testing how pipelines handle upstream schema changes without breaking downstream Gold tables.
- Governance & Security Testing:
- Unity Catalog Validation: Testing Row-Level Security (RLS) and Column-Level Masking to ensure unauthorized users cannot see sensitive data.
- Data Lineage: Validating that data lineage in Unity Catalog correctly reflects the movement of data across the Lakehouse.
About SHELL INFOTECH PTE. LTD.
Similar Jobs
Application Support Analyst
SHELL INFOTECH PTE. LTD.
SGD 48,000 - SGD 72,000/yr
DevSecops Engineer
SHELL INFOTECH PTE. LTD.
SGD 96,000 - SGD 144,000/yr
Data Engineer
SHELL INFOTECH PTE. LTD.
SGD 84,000 - SGD 120,000/yr
Data Engineer
SHELL INFOTECH PTE. LTD.
SGD 96,000 - SGD 144,000/yr
Senior Data Engineer (Databricks)
SHELL INFOTECH PTE. LTD.
SGD 93,600 - SGD 144,000/yr