UX Data Scientist

The Opportunity

 

The Skills You'll Have

• Thrive in fast-moving environments and deliver high-impact outcomes at pace
• Independently identify ambiguous challenges and take ownership of driving them to resolution
• Champion new ideas, influence stakeholders, and rally support around strategic initiatives
• Identify workflows where AI tools and emerging technologies accelerate delivery, and make
  building the default way you work
  

Measurement, Experimentation, and Evaluation

• Define what "good" means for an agentic experience and turn it into metrics, evals, and datasets that hold up under scrutiny
• Design and analyze experiments — A/B tests, holdouts, shadow runs — with correct sizing, clean reads, and honest conclusions
• Build offline and online evaluation systems for LLM behavior across correctness, grounding, safety, tone, and latency
• Apply statistical and ML fluency to real, messy production behavior — distinguishing signal from noise and root cause from symptom

Agentic UX and Behavior Design

• Turn complex enterprise workflows into agent experiences people can actually use without training
• Design context architecture, decision paths, and communication patterns, and reason about how they shape behavior across the full input distribution
• Work directly with customers — running pilots, mining usage, and iterating based on how they actually behave, including customers with complex needs or in delicate situations
• Make tradeoff decisions across user needs, agent behavior, technical constraints, and business goals, and connect those decisions to commercial outcomes

Competitive & Market Awareness

• Maintain running awareness of the competitive landscape across multiple verticals
• Identify gaps and opportunities and produce analysis that connects to product and behavior decisions
• Track market trends, adjacent products, and emerging tools, connecting what's happening in the market to what Sitetracker should be doing
• Regularly contribute to competition and industry Slack channels with substantive observations

Within 90 Days, You'll:

• Collaborate regularly with product management, design, and engineering peers, and have met key stakeholders across the departments you work with

• Be on the phone with customers, learning their workflows, pain points, and how they use Sitetracker and Scout. Know the personas and subverticals

• Deliver work to your scrum team — pushing code, building instrumentation and evals, and demonstrating the builder standard

• Be able to confidently demo Sitetracker's core product in your area and give a solid cross-product demo

• Stand up baseline evals and quality tracking for your area in Watch, and establish what you don't yet measure

• Begin mentoring others and providing substantive feedback in reviews

Within 180 Days, You'll:

• Lead the successful delivery of high-impact agent experiences across complex enterprise workflows

• Own day-to-day decisions for your experience area independently, establishing leading indicators and making tradeoff decisions confidently

• Architect Scout graphs and design sophisticated AI interactions, owning a comprehensive eval suite and tracking quality over time

• Design and run experiments on real agent behavior, and run customer betas and previews independently — including with customers who have more complex needs or stronger opinions

• Actively mentor others and raise the bar through consistent, high-quality reviews of evals, experiments, and agent designs

• Contribute meaningfully to shared evaluation infrastructure and establish higher standards for quality measurement in your domain

Within 365 Days, You'll:

• Point to product successes you have driven that have directly contributed to commercial outcomes

• Be recognized as an expert in your domain — people across the org come to you when they need to understand your experience area or how its quality is measured

• Ship code confidently to production, build the instrumentation and evals that keep quality honest, and operate as a true peer with engineering

• Be trusted with the most complex customer situations and maintain relationships with key customers that you built and maintain on your own

• Shape direction across multiple projects with well-reasoned, evidence-backed positions that influence the roadmap

• Set the pace for delivery in your domain and make others faster by reducing ambiguity early

Employment type

Full-Time