LLM / AI Quality Engineer

Role Name - LLM / AI Quality Engineer

Work Location - Serangoon North

Qualifications:
The ideal candidate should possess:

• 3+ years in software testing/QA with strong test methodology and tooling; hands-on API testing and performance testing.
• Programming familiarity (e.g., Python/TypeScript) and experience with CI/CD and version control.
• Cloud basics (AWS/Azure/GCP) and microservices fundamentals.
• Degree / Diploma in CS/IT or equivalent.

Preferred (AI/ML Focus)

• Understanding of ML concepts and MLOps; experience with model validation and monitoring in production.
• Experience with AI-specific security testing and vulnerability assessment.
• Familiarity with evaluation/observability tools (any of): LangSmith, Weights & Biases, RAGAS, TruLens, Promptfoo, DeepEval, Guardrails/LlamaGuard, Presidio; plus OpenTelemetry-style LLM traces.
• Practical exposure to Azure OpenAI/Bedrock/Vertex and model gateways; quota & token accounting know-how.

Tooling & Automation

• Modern automation frameworks (e.g., Playwright, Cypress, Selenium), API test tools (Postman/REST Assured), performance tool (k6/JMeter), and CI/CD integration.  
• Data evaluation pipelines for RAG (embedding validation, filtering, drift detection).

Traits

• Outcome-oriented, high standards; strong communication and collaboration; customer-focused; proficient in written and spoken English.

Telco Context (Nice-to-Have)

• - Experience testing copilots/agents for BSS/OSS, NOC analytics, and enterprise care; ability to tie eval KPIs to CSAT, AHT, FCR, MTTR.

Job Description
What you will do:

As a LLM / AI Quality Engineer, you will lead the end-to-end evaluation of AI applications—LLM features, RAG systems, and multi-agent workflows—to ensure they meet business outcomes, safety requirements, and platform standards. Own test design, execution, and reporting across offline, pre-prod, and in-prod stages, integrating with CI/CD and working closely with product, data, and platform teams.

1) AI/LLM Evaluation & Test Design

- Define evaluation strategies (golden sets, adversarial suites, regressions), pass/fail gates, and SLOs for quality, safety, latency, and cost.

- Establish rubric-based human reviews (usefulness, faithfulness, safety, clarity) and calibrate annotators.

- Instrument LLM-as-judge where appropriate with calibration and spot checks.

2) RAG, Retrieval, & Grounding

- Measure retrieval precision/recall, MRR/nDCG, and answer faithfulness to sources; detect hallucination and citation errors.

- Test chunking, prompt templates, filters, and policy chains; monitor stale/poisoned content.

3) Agentic & Tool-Use Scenarios

- Validate multi-step plans, tool selection, error recovery, retries, and idempotency for functions with side effects.

- Contract-test JSON schemas and structured outputs across services.

4) Non-Functional, Performance & Cost

- Run token-aware load/soak tests (context length, temperature, batching); track p50/p95/p99, throughput, timeouts, cache hit rate, and cost per successful task.

- Recommend optimizations (prompt/policy changes, retrieval tweaks, caching).

5) Security, Privacy & Safety

- Red-team for prompt injection, data exfiltration, indirect injections via retrieved content; validate guardrails pre/post inference.

- Enforce PII controls, data-residency, and compliance checks; align with organizational security testing practices.

6) Observability & CI/CD Integration

- Implement prompt/dataset/version lineage and trace-based evals; automate in CI (pre-merge golden tests, nightly adversarials) with canary/A-B in prod and rollback criteria.

- Produce clear, decision-ready reports with risk assessments and release recommendations.

7) Project Delivery & Collaboration

- Analyze requirements, enhance test plans with additional cases, prepare environments (including cloud), execute tests per plan, and drive defect resolution.

- Provide regular status updates; manage test activities to schedule; support SIT/UAT and production readiness.    

8) Performance, API & Platform Testing (Carry-over)

- Execute API, performance, and load testing for microservices/web services that underpin AI features; integrate automated testing into CI/CD.

9) Team & Standards

- Adopt and improve test standards/methodology; share practices, train teams, participate in peer reviews, and pursue self-directed learning.