Time since we first saw it
Day 22
Up longer than most. Similar jobs usually come down between day 1 and day 15.
From 1,376 similar postings that came down. Coming down is not always a hire.
San Francisco, CA
Live, first seen by us 22 days agoNice to have: long-horizon or agentic evaluation work; psychometrics, item response theory, or human-subjects research; open-source benchmark or dataset contributions. Compensation: $250-400K OTE + Equity. Includes a built-in bonus and equity program, publication freedom, a conference budget, flexible PTO, 15 holidays, 12 weeks of paid parental leave, 401k match, and an education stipend. This is an ideal list. If the role excites you and you do not check every box, we would still love to hear from you.Read the full posting
Design rubrics and test whether they measure what they claim: validity, reliability, and sensitivity.
Build quality signals that put evidence behind every data point we ship.
Design Dscout’s first benchmark: real users, in real context, over long horizons.
Rubrics and QA science Design rubrics and test whether they measure what they claim: validity, reliability, and sensitivity. Measure how reliable human raters are, and when disagreement is noise versus real signal.
Calibrate AI judges against real human judgment, and study when they drift away from it. Build quality signals that put evidence behind every data point we ship. Long-horizon, real-user benchmarks
Long-horizon, real-user benchmarks Design Dscout’s first benchmark: real users, in real context, over long horizons. Multi-week tasks across tools, not one-shot prompts. Define the protocol, task design, scoring, and release. Work with our delivery team on fieldwork.
Hands-on experience with LLM-as-judge, rubric-based evaluation, and preference data.
Strong Python, and the ability to build research prototypes and evaluation pipelines other people run. You use AI coding tools as a real part of your workflow.
Experience running experiments on messy human data, not only curated datasets.
- Published research in evals, preference learning, reward modeling, human-AI interaction, measurement, or agent evaluation. - A PhD, a PhD in progress, or an MS with two or more strong research internships at labs or top industry groups. If you do not have advanced degrees but have a history of creative machine learning research work, this is just as desirable. - Strong applied statistics: inter-rater reliability, confidence intervals, power, and a nose for when a metric misleads.
Dscout is a team of passionate, empathetic, and curious professionals.
In the employer’s words · Read in context
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
Day 22
Up longer than most. Similar jobs usually come down between day 1 and day 15.
From 1,376 similar postings that came down. Coming down is not always a hire.
2 open jobs
Open jobs by the week we first saw them, last 12 weeks.
Hiring recordAdd your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
Models ace benchmarks and still surprise people in the wild. A benchmark says how capable a model is. It does not say whether the model works for the person using it, across weeks, in the mess of real life. That gap is where we work. We serve AI researchers, the product builders who ship alongside them, and the research teams who partner with both. We are a new unit inside an established company: live customers, a mature platform (React, Elixir, Python, Swift, Kotlin; SOC 2 and HITRUST certified), a small senior team, and room to build. We work AI-native. Agents write code, evals gate changes, and we aim to do in months what used to take years. A note from our Chief AI Officer & GM. The hardest AI problems show up when a model meets real people, with their tools, their interruptions, and their goals. Our customers need a partner who can design the answer, build it, and ship it fast. Ground Truth sits where deployment meets research: what we learn in the field shapes the science, and the science shapes what we ship. Come build it with us.
- Hands-on experience with LLM-as-judge, rubric-based evaluation, and preference data. - Strong Python, and the ability to build research prototypes and evaluation pipelines other people run. You use AI coding tools as a real part of your workflow. - Experience running experiments on messy human data, not only curated datasets.
Complete your application on job-boards.greenhouse.io. The employer’s form will show what is required.
Already applied? Track this application
At Dscout, we’re building the most flexible and powerful UX research platform on the market—trusted by the world’s top brands in finance (JP Morgan Chase, Intuit, Charles Schwab, PayPal), healthcare (Aya, Headspace), consumer goods (Keen, Verizon, Target, Northface), and tech (Google, Amazon, Facebook, Meta, Spotify, AirBnB). Our tools help teams deeply understand the humans behind their products, so they can build better ones. We are expanding our smart and driven team and would love for you to join us.
At Dscout, we're building the human-grounded layer that teaches AI to work in the real world and proves it stays aligned there. We are expanding our smart and driven team and would love for you to join us.
Dscout AI Training & Evals builds human data and evals that show whether AI works for real people. For ten years Dscout has captured how people behave in real context, with consent and scientific rigor. Dscout AI Training & Evals points that engine at AI: consented human behavior and judgment, turned into the training data, evals, and benchmarks AI teams trust.
Models ace benchmarks and still surprise people in the wild. A benchmark says how capable a model is. It does not say whether the model works for the person using it, across weeks, in the mess of real life. That gap is where we work.
We serve AI researchers, the product builders who ship alongside them, and the research teams who partner with both. We are a new unit inside an established company: live customers, a mature platform (React, Elixir, Python, Swift, Kotlin; SOC 2 and HITRUST certified), a small senior team, and room to build. We work AI-native. Agents write code, evals gate changes, and we aim to do in months what used to take years.
A note from our Chief AI Officer & GM. The hardest AI problems show up when a model meets real people, with their tools, their interruptions, and their goals. Our customers need a partner who can design the answer, build it, and ship it fast. Ground Truth sits where deployment meets research: what we learn in the field shapes the science, and the science shapes what we ship. Come build it with us.
Why this seat
This is a research-first role focused on questions where the right rubric, benchmark, or method does not yet exist. You will find the gaps, propose bold directions, and take projects from the first hypothesis through experiments and benchmark construction to publication. You will also build: the science has to run inside live customer programs.
What you will do
Rubrics and QA science
Long-horizon, real-user benchmarks
Human judgment as training and eval signal
New research directions
A good first year
How we will hire: a conversation with the Head of AI Research; a research talk on your publication; a design exercise (how would you validate a rubric for a long-horizon task?); a conversation with the GM, forward deployed engineers, and delivery partners.
What we care about most. A strong publication record helps. We care most about whether you can find the question that matters, design a rigorous way to answer it, and move fast enough that the answer is still useful.
What you bring
Nice to have: long-horizon or agentic evaluation work; psychometrics, item response theory, or human-subjects research; open-source benchmark or dataset contributions.
Compensation: $250-400K OTE + Equity. Includes a built-in bonus and equity program, publication freedom, a conference budget, flexible PTO, 15 holidays, 12 weeks of paid parental leave, 401k match, and an education stipend.
This is an ideal list. If the role excites you and you do not check every box, we would still love to hear from you.
New AI Engineer postings at the employers we follow are up 340% over the last 6 weeks compared with the 6 before.
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
We file this job under AI Engineer, by the work the posting describes.
AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.
Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.
An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.
Selected passages from the saved posting. Check the full description for conditions and exceptions.
Nice to have: long-horizon or agentic evaluation work; psychometrics, item response theory, or human-subjects research; open-source benchmark or dataset contributions. Compensation: $250-400K OTE + Equity. Includes a built-in bonus and equity program, publication freedom, a conference budget, flexible PTO, 15 holidays, 12 weeks of paid parental leave, 401k match, and an education stipend. This is an ideal list. If the role excites you and you do not check every box, we would still love to hear from you.
San Francisco, CA
- An incredible and progressive benefits package (for both you and your dependents) to support work/life balance, including flexible PTO, 15 company holidays, 12 weeks of paid parental leave, 401k match, and much more. - An education stipend to support your growth & development, and a remote work stipend. - A company that is open and transparent with our team. You will know what is happening and why it matters.
Dscout participates in the E-Verify program in certain locations, as required by law. Location Flexibility: Dscout is proud to support a remote-first workforce and enable employees to work from almost anywhere. At this time, however, we are unable to hire in the following locations: Montana, Hawaii, Alaska, and Washington DC. NOTE: DSCOUT NEVER CONTACTS JOB APPLICANTS VIA TEXT, MESSENGER, OR OTHER SIMILAR APPLICATIONS. BE AWARE OF PHISHING AND SPOOFING SCAMS, BOTH VIA TEXT AND EMAIL. ONLY RESPOND TO EMAILS FROM DSCOUT.COM
- A company that is open and transparent with our team. You will know what is happening and why it matters. Dscout is an equal-opportunity employer that values diversity. We do not discriminate based on identity, including race, color, religion, national origin or ancestry, sex, gender identity and expression, age, physical or mental disability, pregnancy, veteran or military status, unfavorable discharge from military service, genetic information, sexual orientation, marital status, order of protection status, citizenship status, arrest record or expunged or sealed convictions, or any other legally recognized protected basis under federal, state, or local law. If you need reasonable accommodations for any part of the employment process, please email us at accommodations@dscout.com with the nature of your request and your contact information. We'll do everything possible to ensure you're well-prepared for success during our interview process, while also upholding your privacy, including accommodating any special requests. Please note that only inquiries regarding requests for reasonable accommodation will be responded to from this email address.