AI /LLM Quality Engineer (Ref 105)

As a LLM / AI Quality Engineer, you will lead the end-to-end evaluation of AI applications—LLM features, RAG systems, and multi-agent workflows—to ensure they meet business outcomes, safety requirements, and platform standards. Own test design, execution, and reporting across offline, pre-prod, and in-prod stages, integrating with CI/CD and working closely with product, data, and platform teams. 

Responsibilities

• AI/LLM Evaluation & Test Design:  Define evaluation strategies (golden sets, adversarial suites, regressions), pass/fail gates, and SLOs for quality, safety, latency, and cost.
• RAG, Retrieval, & Grounding: Measure retrieval precision/recall, MRR/nDCG, and answer faithfulness to sources; detect hallucination and citation errors
• Agentic & Tool-Use Scenarios: Validate multi-step plans, tool selection, error recovery, retries, and idempotency for functions with side effects.
• Non-Functional, Performance & Cost: Run token-aware load/soak tests (context length, temperature, batching); track p50/p95/p99, throughput, timeouts, cache hit rate, and cost per successful task.
• Security, Privacy & Safety: Red-team for prompt injection, data exfiltration, indirect injections via retrieved content; validate guardrails pre/post inference.
• Observability & CI/CD Integration: Implement prompt/dataset/version lineage and trace-based evals; automate in CI (pre-merge golden tests, nightly adversarials) with canary/A-B in prod and rollback criteria.
• Project Delivery & Collaboration: Analyze requirements, enhance test plans with additional cases, prepare environments (including cloud), execute tests per plan, and drive defect resolution.
• Performance, API & Platform Testing (Carry-over) Execute API, performance, and load testing for microservices/web services that underpin AI features; integrate automated testing into CI/CD. 

Requirements:

• 3+ years in software testing/QA with strong test methodology and tooling; hands-on API testing and performance testing. 
• Programming familiarity (e.g., Python/TypeScript) and experience with CI/CD and version control. 
• Cloud basics (AWS/Azure/GCP) and microservices fundamentals. 
• Tooling & Automation: Modern automation frameworks (e.g., Playwright, Cypress, Selenium), API test tools (Postman/REST Assured), performance tools (k6/JMeter), and CI/CD integration.   

Kindly indicate 105 – AI Quality Engineer on the subject

Interested applicants may send in their detailed resumes, with subject title: 105– AI Quality Engineer in MS Word format

We regret that only shortlisted candidates will be notified.

Formilleza Jacqueline Rey (Reg. Nos. R1104128)
IT Consultancy & Services Pte Ltd
EA License No.: 92C5311
For more information about ITC, please visit us @ www.itcspl.com