QA Engineer, AI

The Mortgage Office•San Mateo, CA

20h•Onsite

About The Position

We are hiring for a QA Engineer, (AI), who will be responsible for validating both: Deterministic systems (APIs, backend logic, UI flows) and Non-deterministic AI systems (LLM outputs, prompt behavior, edge cases). We are building an AI-first product where core functionality depends on prompt engineering and LLM behavior. This position requires someone to design, execute, and automate testing strategies for LLM-based features, and AI-driven workflows. Additionally, they will test end-to-end product flows across frontend, backend, and APIs.

Requirements

3+ years in QA / software testing, preferably for LLM based product
Strong Python skills (Automation, LLM, Evaluations)
Experience testing APIs and full-stack systems
Ability to think in edge cases and break systems intentionally
Experience working with LLM-based products
Experience testing prompt engineering products

Responsibilities

Test end-to-end product flows across frontend, backend, and APIs of an event based system
Design structured test cases for prompt-based features
Build and maintain test datasets
Define evaluation criteria for AI outputs
Run regression testing when prompts or models change• Automate repeatable tests
Track quality metrics over time (output quality, failure rate, latency issues)

Benefits

Comprehensive health, dental, and vision coverage.
401(k) with company funded match.
Mission-driven company with a 45+ year legacy and industry-leading products trusted by thousands of lenders.
Collaborative, supportive team known for long tenure, deep expertise, and a culture built on trust and integrity.
Help modernize and evolve the leading platform in private lending technology.
Beautiful San Mateo office with a tight-knit culture.
Competitive compensation and room for professional growth.
Performance-based bonuses.
Generous PTO.
Modern workspace with on-site amenities.
Awesome company Christmas party.