(470) 443-4800Get a demo
All roles

Engineering

AI Evals and Quality Engineer

Full-timeRemoteReports to: Founder and Chief Executive Officer

Turn every approval, rejection and fix our people make into tests that keep our agents right.

Apply

About SpareBrains

SpareBrains helps small and medium-sized businesses run better. A team of people and AI agents does real work for them. It finds where they’re losing customers online, fixes it, and keeps it running week after week. Our agents work inside our clients’ own accounts and tools, always with the owner’s approval. That makes trust the heart of what we sell.

The role

Every time a person approves or rejects an agent’s work, we learn something. You’ll turn that into evals: test sets, graders and gates that catch problems before owners do. Your work means our people review less and our agents get better. In your application, tell us about a failure you caught that others missed.

What you’ll do

Build the evals

  • Build golden test sets from real failures: a few dozen strong cases per agent, not thousands of made-up ones.
  • Write graders: fixed checks first, then AI judges checked against human ratings.
  • Add a gate that blocks prompt or tool changes when quality drops.

Watch production

  • Score live agent runs, and flag drift early.
  • Turn every failure caught in production into a permanent test.
  • Compare models on accuracy, speed and cost.

Work with the team

  • Turn our operations team’s reviews into labels.
  • Help engineers read transcripts and find the real cause.

What you’ll bring

  • 3+ years in software or machine learning engineering.
  • Strong Python or TypeScript.
  • A feel for statistics: variance, sample size and agreement between reviewers.
  • Hands-on use of an eval or tracing tool.
  • The habit of reading transcripts, not only dashboards.

You don’t need every item on this list. If the work excites you, apply anyway.

Nice to have

  • Voice evals: latency, interruptions and transcription accuracy.
  • Red-teaming AI systems.
  • Knowing when to automate a check and when a person should decide.

What we offer

A competitive base salary, equity, and the chance to build SpareBrains from its first day.

How to apply

Send your name, your email and your resume, with links to your work if you like. No cover letter needed.

Apply

Interview process

  1. Intro call. 30 minutes with our founder on the role, your work and your questions.
  2. Role interview. A deeper conversation about how you’d do the job, and the work you’ve done.
  3. Working session. We work through a real problem together, with any tools you like, AI included.
  4. References and offer. We speak to people you’ve worked with, then send the offer in writing.