Back to jobs

Senior AI Engineer, Evaluation & Optimization

Reads like an ML Engineer role: training models and running them in production.

Pay
$200,000–260,000/year · Base — pay source
We’re playing offense to win! Our product market fit and our world-class employees make us the leader in our space. We’re building something cool and people like it here. The base salary range for this position is $200,000-$260,000/year. Final compensation will be determined based on factors including experience and qualifications. Vantaca offers comprehensive benefits including medical, dental, and vision insurance through Blue Cross Blue Shield of North Carolina and Principal; Health Savings and Flexible Spending Accounts; a 401(k) plan with company match (up to 3% after one year, with Vantaca covering all administrative fees); and at no cost to you: life and short term disability insurance, telemedicine through Teladoc, and an Employee Assistance Program providing 24/7 access to counseling, financial planning, legal services, and family support resources. We receive many resumes for our open positions and each one is reviewed by a human being on our recruiting team. We will compare your background with the qualifications and requirements for the position.
Read the full posting
Work setting
Hybrid — work setting source
Work setup in the employer’s posting: Hybrid
From the employer’s posting
Employment
Full-time — employment source
Employment type in the employer’s posting: FULL_TIME
From the employer’s posting

What you’ll work on

Full posting

We run millions of AI tasks in production and are hiring a Senior AI Engineer, Evaluation & Optimization to own the systems, experiments, and feedback loops that determine how those tasks use AI models.

AI models are improving rapidly, but model capability alone does not determine production performance.

  • You will build evaluation and observability infrastructure, benchmark models and configurations, analyze production behavior, and translate findings into production improvements.

From the employer’s posting
We run millions of AI tasks in production and are hiring a Senior AI Engineer, Evaluation & Optimization to own the systems, experiments, and feedback loops that determine how those tasks use AI models.
AI models are improving rapidly, but model capability alone does not determine production performance. Model choice, prompts, context, tool use, routing, retries, and execution design all affect whether a task succeeds and what that success costs.
AI models are improving rapidly, but model capability alone does not determine production performance. Model choice, prompts, context, tool use, routing, retries, and execution design all affect whether a task succeeds and what that success costs. Your mandate is to maximize successful task completion per dollar while meeting our quality, reliability, and latency requirements. You will build evaluation and observability infrastructure, benchmark models and configurations, analyze production behavior, and translate findings into production improvements. You will also design optimization loops that search for better prompts, model configurations, routing policies, agent strategies, and execution plans. This is an engineering-heavy, highly empirical role. You will work across model providers, prompts, agent architectures, routing strategies, production traces, and optimization systems to determine what works at scale for our workloads. TypeScript is the primary implementation language.

Before you apply

Employer’s visa record
3 H-1B petitions approved in fiscal 2024 and 2025For the employer as a whole. This posting doesn’t say whether the role is sponsored. US Citizenship and Immigration Services.

What you’ll bring

All qualifications

Core experience

  • Experience building production systems that use LLMs or other machine-learning models.
  • Experience operating high-volume or high-reliability services.
  • Experience designing and running rigorous model evaluations.
  • Strong understanding of modern LLM APIs and model behavior.
  • Experience working with production traces, datasets, and experimentation infrastructure.
  • Ability to reason quantitatively about quality, latency, reliability, and cost.

Preferred experience

  • Experience in one or more of the following areas is helpful:
  • Experience with CUDA, GPU kernels, vLLM, TensorRT-LLM, or low-level inference optimization is useful but not required.
Qualification wording
Experience building production systems that use LLMs or other machine-learning models.
Experience operating high-volume or high-reliability services.
Experience designing and running rigorous model evaluations.
Strong understanding of modern LLM APIs and model behavior.
Experience working with production traces, datasets, and experimentation infrastructure.
Ability to reason quantitatively about quality, latency, reliability, and cost.
Experience in one or more of the following areas is helpful:
Experience with CUDA, GPU kernels, vLLM, TensorRT-LLM, or low-level inference optimization is useful but not required.
Apply at Vantaca

Posting record

Report an error
Status in our records
Live
First seen by us
Oct 18, 202630 days ago
Recorded sightings
30
Last seen by us
Nov 17, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Pay

In line with similar jobs

$200K–$260K against $166K–$237K, the middle half of what similar US postings state.

From 1,505 postings in the last year. Stated ranges, not salaries paid.

Time since we first saw it

Day 30

Similar jobs usually come down between day 7 and day 76.

From 5,756 similar postings that came down. Coming down is not always a hire.

Vantaca is hiring

2 open jobs

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • SQL
  • TypeScript
  • Node.js
Where the posting names them
- Ability to keep systems reliable while their prompts, models, or execution policies change continuously. - Python experience is useful for analysis and evaluation tooling, but TypeScript is the primary language for production systems in this role. Preferred Experience
- Ability to reason quantitatively about quality, latency, reliability, and cost. - Strong SQL and data-analysis skills. - Experience designing controlled experiments and interpreting noisy results.
Your mandate is to maximize successful task completion per dollar while meeting our quality, reliability, and latency requirements. You will build evaluation and observability infrastructure, benchmark models and configurations, analyze production behavior, and translate findings into production improvements. You will also design optimization loops that search for better prompts, model configurations, routing policies, agent strategies, and execution plans. This is an engineering-heavy, highly empirical role. You will work across model providers, prompts, agent architectures, routing strategies, production traces, and optimization systems to determine what works at scale for our workloads. TypeScript is the primary implementation language. What You Will Own
- Build the infrastructure required to measure, operate, and improve millions of AI tasks reliably. - Build TypeScript services and libraries for model execution, routing, retries, fallbacks, and structured outputs. - Instrument AI workflows for quality, latency, reliability, token usage, and cost.
Requirements - Strong software engineering skills in TypeScript and Node.js. - Experience building production systems that use LLMs or other machine-learning models.

Your next step

Complete your application on jobs.gem.com. The employer’s form will show what is required.

Already applied? Track this application

Job description

View original posting ↗

HOAi is the fastest-growing company revolutionizing the community association management industry. Our AI workforce platform integrates machine learning technology to streamline labor-heavy processes, eliminating inefficiencies and driving scalability. With rapid growth in the AI space, we are pushing boundaries to redefine industry standards. HOAi Agents go beyond traditional AI by proactively executing complex, multi-step processes with human-like reasoning—working autonomously, 24/7, across your entire operation. This transformation optimizes labor costs, enables growth without additional hires, and ensures faster, higher-quality service for residents and board members.

HOAi was acquired by Vantaca in the fall of 2024. Vantaca just achieved unicorn status with a $1.25B valuation, so it's safe to say we're past the "scrappy startup phase." We're not just building a successful company – we're building the category-defining platform that will transform how an entire industry operates. 
 
Here's the reality of our trajectory:
  • Growing 100% year-over-year
  • Our AI product (HOAi) went from $0 to millions in months
  • Backed by Cove Hill Partners and JMI Private Equity
  • 6M+ doors on our platform, displacing legacy systems

About the Role


We run millions of AI tasks in production and are hiring a Senior AI Engineer, Evaluation & Optimization to own the systems, experiments, and feedback loops that determine how those tasks use AI models.

AI models are improving rapidly, but model capability alone does not determine production performance. Model choice, prompts, context, tool use, routing, retries, and execution design all affect whether a task succeeds and what that success costs.

Your mandate is to maximize successful task completion per dollar while meeting our quality, reliability, and latency requirements. You will build evaluation and observability infrastructure, benchmark models and configurations, analyze production behavior, and translate findings into production improvements. You will also design optimization loops that search for better prompts, model configurations, routing policies, agent strategies, and execution plans.

This is an engineering-heavy, highly empirical role. You will work across model providers, prompts, agent architectures, routing strategies, production traces, and optimization systems to determine what works at scale for our workloads. TypeScript is the primary implementation language.

What You Will Own


Production AI Infrastructure


  • Build the infrastructure required to measure, operate, and improve millions of AI tasks reliably.
  • Build TypeScript services and libraries for model execution, routing, retries, fallbacks, and structured outputs.
  • Instrument AI workflows for quality, latency, reliability, token usage, and cost.
  • Create consistent interfaces across model providers and configurations.
  • Provide end-to-end traceability across prompts, models, tools, agent steps, retries, and outcomes.
  • Define durable schemas for tasks, evaluations, experiments, model versions, and production results.
  • Design systems for safe rollout, rollback, and comparison of model and workflow changes.
  • Improve the reliability and operational efficiency of high-volume AI workloads.
  • Partner with platform and product engineers so teams can adopt optimization capabilities throughout the stack.
  • This is a production engineering role. The systems you build will run in live, high-volume production environments.

Model Evaluation


  • Build and maintain evaluation suites that represent our production workloads and predict real production performance.
  • Define task-specific quality metrics and acceptance thresholds.
  • Build golden datasets from production workloads, human-reviewed examples, edge cases, and synthetic scenarios.
  • Develop deterministic checks, automated graders, and LLM-as-judge evaluations where each method is appropriate.
  • Calibrate automated evaluations against human judgment.
  • Build regression tests that detect degradation as models, prompts, tools, and agent architectures change.
  • Sample and analyze production tasks to identify new failure modes.
  • Continuously update evaluations as production behavior changes.
  • Connect offline evaluation results to online task, user, and business outcomes.

Model Experimentation


  • Continuously evaluate the model ecosystem and the configurations around it, including: Models from OpenAI, Anthropic, Google, open-weight providers, and emerging vendors.
  • Model sizes, capability tiers, reasoning effort, and inference-time compute.
  • Prompt, system-instruction, context-selection, and context-compression strategies.
  • Tool-use configurations, structured outputs, and agent architectures.
  • Retries, fallbacks, escalation policies, and model cascades.
  • Execution parallelism, task decomposition, and planning approaches.
  • Experiments should be reproducible, statistically sound, and connected to production outcomes. They should produce clear recommendations and deployable changes rather than one-off benchmark results.

Continuous Optimization


  • Build and operate systems that continuously search for better AI configurations across prompts, model selection, routing, reasoning budgets, context construction, tool use, agent workflows, retries, caching, parallelism, and workflow-specific policies.
  • Select the appropriate search method for each problem, from manual experimentation and grid or random search to Bayesian optimization, bandit methods, evolutionary approaches, and production feedback loops.
  • Optimization must be safe and measurable. Build guardrails against flawed graders, narrow-set overfitting, segment-level regressions, reward hacking, and hidden operational costs.
  • Candidate generation and evaluation pipelines.
  • Experiment queues and prioritization.
  • Champion and challenger comparisons.
  • Automatic promotion and rollback criteria.
  • Exploration and exploitation policies.
  • Segment-level monitoring and regression detection.
  • Human-review escalation for ambiguous or high-impact changes.
  • Versioning for prompts, models, tools, datasets, graders, and optimization policies.

Cost and Performance Optimization


  • Establish the Pareto frontier across quality, cost, latency, and reliability for each major workload. Investigate opportunities such as: Use smaller models where larger models provide little incremental value.
  • Route requests dynamically based on task difficulty.
  • Reduce unnecessary context, output tokens, agent steps, tool calls, retries, and model calls.
  • Tune reasoning and token budgets.
  • Compress prompts and context without losing required information.
  • Use caching, reuse, model cascades, fallbacks, and execution parallelism where they improve economics.
  • Compare fine-tuning or distillation with frontier-model APIs where appropriate.
  • Improve provider selection and workload placement.
  • Reduce operational overhead in high-volume execution paths.
  • The objective is to minimize cost per successful task while preserving the quality and reliability users require.

Production Deployment and Monitoring


  • Turn experimental results into measurable improvements in production.
  • Build dashboards for quality, success rate, latency, reliability, and AI economics by workflow, model, configuration, and customer segment.
  • Run controlled production experiments and A/B tests.
  • Roll out model migrations and configuration changes safely.
  • Detect regressions after model or provider updates.
  • Establish automated gates for model, prompt, tool, and agent changes.
  • Monitor optimization systems for reward hacking and grader exploitation.
  • Identify where production behavior diverges from offline evaluation.
  • Partner with product and engineering teams to determine where additional model capability creates meaningful value.
  • Quantify optimization results in successful tasks, user outcomes, and spend.

What Success Looks Like


Within your first several months, the team should be able to answer questions such as:
  • What does each major AI workflow cost, and what quality threshold does it require?
  • Which model and configuration provide the best economics for each workload?
  • How much additional quality do we gain from incremental spend?
  • Can most of a workload move to a substantially less expensive model without material quality loss?
  • Which tasks genuinely require frontier models?
  • Which prompts, workflows, and routing policies are improving over time?
  • Can we improve continuously without overfitting or degrading important segments?
  • When a new model launches, can we determine within days whether and where to adopt it?
  • Can we change models without unknowingly degrading production quality?
  • Can we detect and roll back a harmful optimization automatically?
  • Do our evaluations and graders remain aligned with real user and business outcomes?
  • Over time, optimization should become a repeatable operating loop: production traces reveal failure modes; failure analysis generates candidates; evaluations identify promising changes; controlled rollouts validate them; and production feedback informs the next iteration.

Requirements


  • Strong software engineering skills in TypeScript and Node.js.
  • Experience building production systems that use LLMs or other machine-learning models.
  • Experience operating high-volume or high-reliability services.
  • Experience designing and running rigorous model evaluations.
  • Strong understanding of modern LLM APIs and model behavior.
  • Experience working with production traces, datasets, and experimentation infrastructure.
  • Ability to reason quantitatively about quality, latency, reliability, and cost.
  • Strong SQL and data-analysis skills.
  • Experience designing controlled experiments and interpreting noisy results.
  • Experience building optimization, search, experimentation, or feedback-loop systems.
  • Ability to translate experimental findings into production engineering changes.
  • Sound judgment about when benchmark differences matter and when they do not.
  • Ability to keep systems reliable while their prompts, models, or execution policies change continuously.
  • Python experience is useful for analysis and evaluation tooling, but TypeScript is the primary language for production systems in this role.

Preferred Experience


Experience in one or more of the following areas is helpful:
  • Model routing, cascading, or agent evaluation.
  • LLM-as-judge systems and human evaluation.
  • Online experimentation and A/B testing.
  • Bandits, Bayesian optimization, evolutionary search, or other optimization methods.
  • Automated prompt optimization, reward modeling, or feedback-loop design.
  • Guarding against reward hacking and grader exploitation.
  • Fine-tuning, distillation, or open-weight models.
  • AI observability, tracing, prompt optimization, or context optimization.
  • Semantic caching or prefix caching.
  • High-volume distributed systems, queues, scheduling, rate limits, or provider capacity management.
  • Internal benchmarking, experimentation, AI infrastructure, or inference platforms.
  • Experience with CUDA, GPU kernels, vLLM, TensorRT-LLM, or low-level inference optimization is useful but not required.
  • This role focuses on how we use models and operate AI workloads rather than on the internals of model-serving infrastructure.

Example Problems


Model selection A frontier model costs eight times more than a smaller model. Determine which production tasks receive enough quality improvement to justify the additional cost, then implement routing that sends only those tasks to the frontier model.

Reasoning optimization Determine whether additional reasoning effort materially improves task completion, establish the right reasoning budget for each workload, and continue testing whether the policy can be improved.

Model routing Design a policy that sends routine tasks to inexpensive models and escalates difficult tasks without materially reducing quality or increasing tail latency.

Agent optimization A workflow makes 14 model calls. Identify which calls are necessary, where smaller models can be substituted, which calls can run in parallel, and whether the workflow can complete reliably in fewer steps.

Prompt optimization Generate and evaluate prompt candidates against representative production tasks. Promote improvements that generalize across important segments and roll back candidates that degrade online outcomes.

Workflow optimization A production workflow succeeds 82 percent of the time at a known cost and latency. Build an optimization loop that explores alternative prompts, models, tool sequences, and retry policies while preserving quality thresholds and monitoring for grader exploitation.

New model evaluation A new model launches tomorrow. Run the relevant production evaluations, compare its quality, cost, latency, and reliability with the current configuration, and recommend where, if anywhere, to adopt it.

Production regression A provider silently changes model behavior. Detect the regression, identify affected workflows and customer segments, and roll back or reroute traffic while the team investigates.

Who will be Successful in this Role


  • You instinctively ask, “How do we know, and how do we improve it continuously?” You care less about which model is considered best than about which model and configuration are best for a specific workload, quality threshold, price, and operating scale.
  • You are comfortable moving between production engineering, experimentation, data analysis, optimization, and product judgment. You treat AI quality as an operational property that must be measured, monitored, and improved continuously.
  • You want direct ownership of the infrastructure and feedback loops that determine the performance and economics of millions of production AI tasks.

What We Value


  • Extreme ownership: Takes full responsibility for outcomes, not just inputs 
  • Customer-focused: Understands how infrastructure decisions impact end-user experience 
  • Data-driven: Makes decisions based on metrics and evidence 
  • Continuous learner: Stays current with evolving technologies and best practices 
  • Quality-focused: Builds systems that are reliable, maintainable, and elegant 
  • Velocity-minded: Balances speed with quality; ships incrementally and iterates 
  • Bar raiser: Sets and maintains high standards; elevates team performance and output quality 
  • Strong delegator: Empowers others effectively; distributes work based on strengths and growth opportunities 

Why You Should Join Our Team


  • Our eNPS is +68! (Google it, that is great).
  • Benefits: Medical, Dental, and Vision kick in day one.
  • Unlimited PTO (with a requirement for employees to take a minimum of one continuous week per year).
  • 401K with Company Match.
  • Remote Flexible - come to the office when needed.
  • Great parental leave benefits.
  • Fortune Best Workplaces in Technology™ 2026 (Small/Mid).
  • Named on Inc 5000 list of America’s Fastest Growing Private Companies.
  • Named on Inc 5000 Vet 100 Private Companies list multiple years in a row.
  • Winner of Coastal Entrepreneur Award, Technology Category.
  • Active employee-led Culture Committee.
  • Ongoing industry and professional development trainings available to all employees.
  • Multiple leaders on the executive committee recognized as 40 under 40 recipients for contributions to business and community.
  • We’re playing offense to win! Our product market fit and our world-class employees make us the leader in our space. We’re building something cool and people like it here.

The base salary range for this position is $200,000-$260,000/year. Final compensation will be determined based on factors including experience and qualifications. Vantaca offers comprehensive benefits including medical, dental, and vision insurance through Blue Cross Blue Shield of North Carolina and Principal; Health Savings and Flexible Spending Accounts; a 401(k) plan with company match (up to 3% after one year, with Vantaca covering all administrative fees); and at no cost to you: life and short term disability insurance, telemedicine through Teladoc, and an Employee Assistance Program providing 24/7 access to counseling, financial planning, legal services, and family support resources.

We receive many resumes for our open positions and each one is reviewed by a human being on our recruiting team. We will compare your background with the qualifications and requirements for the position.

If you are selected for an interview you will receive an e-mail from someone on our recruiting team with an @vantaca.com email address. It may take some time for us to review all of the applications so give us some time to respond. We appreciate your interest in this role.

Demand for ML Engineer roles

New ML Engineer postings at the employers we follow are up 12% over the last 6 weeks compared with the 6 before.

Weekly new ML Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 556, the most 1,080, and the latest week had 556.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under ML Engineer, by the work the posting describes.

ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.

Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.

A data scientist often works out which model to build; an ML engineer makes it run reliably in production.

US government figures for computer and information research scientists

Median yearly pay, May 2025
$140,300
Expected change, 2025 to 2035
21.8% more jobsagainst 3.5% for all jobs

The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay
We’re playing offense to win! Our product market fit and our world-class employees make us the leader in our space. We’re building something cool and people like it here. The base salary range for this position is $200,000-$260,000/year. Final compensation will be determined based on factors including experience and qualifications. Vantaca offers comprehensive benefits including medical, dental, and vision insurance through Blue Cross Blue Shield of North Carolina and Principal; Health Savings and Flexible Spending Accounts; a 401(k) plan with company match (up to 3% after one year, with Vantaca covering all administrative fees); and at no cost to you: life and short term disability insurance, telemedicine through Teladoc, and an Employee Assistance Program providing 24/7 access to counseling, financial planning, legal services, and family support resources. We receive many resumes for our open positions and each one is reviewed by a human being on our recruiting team. We will compare your background with the qualifications and requirements for the position.
Location & working pattern

Palo Alto, CA

- 401K with Company Match. - Remote Flexible - come to the office when needed. - Great parental leave benefits.
Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.