Research Engineer

Fluency•San Francisco, CA

2d•Onsite

About The Position

Fluency is looking for a Research Engineer to design experiments, build evaluation infrastructure, and drive model quality for our process conformance, productivity measurement, and AI impact analysis across Fortune 500 organisations. The Problem Space You'll be developing the methodology and systems that determine whether our models actually work. Screenshots, OCR text, application metadata, behavioural signals: the inputs are messy and the ground truth is ambiguous. The challenge is building rigorous evaluation frameworks that quantify model performance and identify improvement opportunities. This means: Designing evaluation pipelines that measure accuracy, precision, and recall across classification tasks Building ground truth datasets from ambiguous, real-world enterprise data Running systematic prompt engineering experiments to optimise LLM performance Developing A/B testing infrastructure for model comparison Researching novel approaches to process understanding, activity classification, and intent extraction Quantifying cost-accuracy tradeoffs across different model architectures and prompting strategies The playbook doesn't exist. You'll write it. You'll work directly with founders and our engineering team on technical challenges that span LLM evaluation, experimental design, and applied research.

Requirements

Strong Python fundamentals and software engineering discipline
LLM prompt engineering and optimisation (token efficiency, few-shot design, chain-of-thought)
Experience evaluating model performance: accuracy measurement, error analysis, regression detection
Ability to read, synthesise, and apply ML research papers
Statistical literacy: understanding when results are meaningful vs noise
Comfort with ambiguity and novel problem domains
Computer Science Background, with caveat. If you don't have a CS background, you're challenged to beat one of the founders in a 1:1 whiteboard duel on DS&A judged by Hung. Neither founder has a formal CS background, but come prepped.
There will be an expectation to stay up to business context, which could involve: Watching key customer calls Interacting with customers Helping with product thinking

Nice To Haves

Experience building evaluation frameworks and benchmarking systems
Ground truth dataset creation and annotation pipeline experience
Experience with hybrid LLM/rule-based systems
OCR, document understanding, or computer vision background
Cost optimisation for LLM-heavy systems
Classification and NLP systems experience
Published research or formal research methodology training
Familiarity with process mining or workflow analysis
Interesting personal projects that demonstrate depth

Responsibilities

Designing evaluation pipelines that measure accuracy, precision, and recall across classification tasks
Building ground truth datasets from ambiguous, real-world enterprise data
Running systematic prompt engineering experiments to optimise LLM performance
Developing A/B testing infrastructure for model comparison
Researching novel approaches to process understanding, activity classification, and intent extraction
Quantifying cost-accuracy tradeoffs across different model architectures and prompting strategies