Applied AI Engineer, Evaluations Intern
The intern will construct and refine benchmark datasets for evaluating LLM-driven patent extraction pipelines, using tools like Claude and Antigravity. They will also run blind‑test evaluations to measure extraction reliability and support stakeholder‑facing deployments.
Skills required
About this role
NUSX Labs is NUSX's technical arm, turning NUS research into inventions, patents, and licensed deals. We're a small, hands-on team, and every AI system we ship is graded against a ground-truth dataset built by hand: for six months, you're the one building it, and your judgments become the benchmark the next model is measured against. It's judgment work, not data entry, across an LLM pipeline that reads patent documents into structured data, a natural-language query layer, and early-signal deal-sourcing tools. You'll be supervised directly by an engineer on the team, with real stakeholder-facing projects and deployment experience.
Roles and Responsibilities
- Build benchmark datasets used to evaluate model performance, across approximately 23 document types, using Claude and Antigravity to draft and an annotation tool to review and refine.
- Run the blind-test evaluation matrix on vendor deliverables and measure extraction reliability against the ground truth.
- Identify edge cases, structural mismatches, and hallucinations that surface-level review would miss, and document them.
- Review existing system interfaces to determine what each screen stores, and build the gold field and schema catalog.
- Compile findings into quality-acceptance reports used by the team to sign off on deliveries.
- Scope may extend beyond the responsibilities listed above.
Requirements
- Demonstrated experience building and shipping a data pipeline (personal, academic, or hackathon) to completion.
- Working knowledge of Python for processing structured data (JSON/CSV). • Familiarity with evaluation concepts (precision, recall, accuracy) and LLM evaluation metrics.
- Willingness to learn domain knowledge (e.g. patents).
- Basic familiarity with version control (git).
- Willingness to use AI tools (Claude, Antigravity) daily.
- Nice to have: basic SQL; basic cloud experience (GCP/AWS); familiarity with data engineering concepts; comfortable reading and working in English and Chinese.