Back to jobs

Product Analyst

Pay
Pay not listed
Work setting
On-site — work setting source
Work setup in the employer’s posting: IN_OFFICE
From the employer’s posting
Employment
Full-time — employment source
Employment type in the employer’s posting: FULL_TIME
From the employer’s posting

What you’ll work on

Full posting
  • Run LLM benchmarking comparisons across model providers for post-call extraction

  • Build and maintain LLM-as-judge eval pipelines and inter-rater reliability analysis

  • Evaluate WER and related quality metrics across ASR providers for Indic languages

  • Analyze self-serve funnel drop-off from signup to expansion

  • Build repeatable dashboards and scripts for ongoing analysis cadence

From the employer’s posting
Run structured comparisons across model providers (e.g. Sarvam, DeepSeek, Gemini, Claude variants) for tasks like post-call extraction and LLM-as-judge scoring
Build and maintain LLM-as-judge pipelines (e.g. using DeepEval), design and track eval metrics, and run inter-rater reliability analysis (e.g. Krippendorff's alpha) across human call reviewers.
Evaluate WER and related quality metrics across ASR providers and models for Indic languages
Instrument and analyze the self-serve funnel (signup → activation → habit → expansion), identifying where users drop off and surfacing friction points for the product team to act on.
Build repeatable dashboards/scripts (not one-off notebooks) so these analyses run as an ongoing cadence, not a one-time project

Before you apply

New graduates
Open to new graduates0–2 years of experience - new grad or early-career, in a data/product analyst, applied ML, or research-adjacent role

About Bolna

Bolna is a YC-backed voice AI orchestration platform built for the Indian market - powering multilingual, vernacular voice agents (Hindi, Hinglish, Tamil, and 10+ languages) at sub-500ms latency across collections, recruitment, sales, and e-commerce use cases.

In the employer’s words · Read in context

Apply at Bolna

Posting record

Report an error

Open 3 monthsWe first saw this posting 116 days ago and it is still up. Roles open this long are often slow to fill or kept open to collect applications.

Status in our records
Live
First seen by us
Jul 29, 2026116 days ago
Recorded sightings
58
Last seen by us
Nov 20, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 116

Up longer than most. Similar jobs usually come down between day 5 and day 43.

From 16,008 similar postings that came down. Coming down is not always a hire.

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • SQL
  • Azure
  • pandas
Where the posting names them
- 0–2 years of experience - new grad or early-career, in a data/product analyst, applied ML, or research-adjacent role. We're hiring for raw analytical strength and trainability, not a finished track record. - Strong SQL and Python (pandas at minimum) from coursework, internships, or prior work - comfortable writing and debugging your own queries and scripts without hand-holding. - Solid statistical fundamentals (distributions, basic hypothesis testing, agreement/reliability concepts) - you don't need production experience with inter-rater reliability metrics, but you should pick up a new stats concept quickly when pointed at it.
- ASR/voice benchmarking: Evaluate WER and related quality metrics across ASR providers and models for Indic languages; work with public benchmarks and academic references where relevant. - Infra & latency analytics: Analyze routing, latency, and cost data (e.g. Azure PTU utilization, percentile latency distributions) to inform infrastructure and routing decisions. - Agent behavior analytics: Support population-level analysis of graph agent behavior - node-level aggregates, designed-vs-observed graph diffs, stuck-in-loop detection and similar failure-pattern metrics.

Your next step

Complete your application on jobs.gem.com. The employer’s form will show what is required.

Already applied? Track this application

Job description

View original posting ↗

Reports to: Head of Product
Team: Product - a horizontal role supporting whichever pod needs rigorous analysis in a given week (Observability, Workflows, Graph Agents, Infra), with work assigned and prioritized weekly rather than owned by a single pod



About Bolna


Bolna is a YC-backed voice AI orchestration platform built for the Indian market - powering multilingual, vernacular voice agents (Hindi, Hinglish, Tamil, and 10+ languages) at sub-500ms latency across collections, recruitment, sales, and e-commerce use cases. We're an orchestration layer, not a model company: our moat is outcome-labeled vernacular data, a rigorous evaluation infrastructure, and a growing taxonomy of how Indian enterprise voice AI actually fails in production.



Why this role exists


Product decisions at Bolna increasingly hinge on rigorous, code-mixed-aware data analysis - and not just one kind. On one side there's model/eval rigor: LLM benchmarking for post-call intelligence, ASR/WER evaluation, inter-rater reliability on human-labeled calls, routing/latency economics. On the other there's product and growth insight: understanding where self-serve users drop off in their journey, what patterns emerge across lakhs of monthly calls, which use cases and configs are actually working. Both currently sit with the Head of Product alongside strategy and roadmap ownership. We need a dedicated analyst to own the execution and the recurring cadence across both - freeing product leadership to act on findings rather than produce them.



What you'll do


Model & evaluation analysis


  • LLM & model benchmarking: Run structured comparisons across model providers (e.g. Sarvam, DeepSeek, Gemini, Claude variants) for tasks like post-call extraction and LLM-as-judge scoring - evaluating cost, accuracy, fill rate, and TTR, with particular attention to Hinglish and code-mixed content.
  • Evaluation infrastructure: Build and maintain LLM-as-judge pipelines (e.g. using DeepEval), design and track eval metrics, and run inter-rater reliability analysis (e.g. Krippendorff's alpha) across human call reviewers.
  • Golden dataset creation: Support the construction of golden datasets for ASR/transcript labeling - including flagging conventions like code-switch scripting (Devanagari vs. Roman) and transliteration normalization for decision before scoring is meaningful.
  • ASR/voice benchmarking: Evaluate WER and related quality metrics across ASR providers and models for Indic languages; work with public benchmarks and academic references where relevant.
  • Infra & latency analytics: Analyze routing, latency, and cost data (e.g. Azure PTU utilization, percentile latency distributions) to inform infrastructure and routing decisions.
  • Agent behavior analytics: Support population-level analysis of graph agent behavior - node-level aggregates, designed-vs-observed graph diffs, stuck-in-loop detection and similar failure-pattern metrics.



Product & growth insight generation


  • Self-serve journey analysis: Instrument and analyze the self-serve funnel (signup → activation → habit → expansion), identifying where users drop off and surfacing friction points for the product team to act on.
  • Cross-customer call insights: Mine aggregate call data across customers and use cases for patterns - completion rates by use case, which model/config combinations perform best where, emerging failure patterns across prompt templates - feeding both internal roadmap decisions and future customer-facing benchmarking.
  • Ad hoc product analysis: Be the fast, reliable "pull me the data on X" resource for pod PMs - usage patterns, cohort behavior, feature adoption - turning raw usage data into a clear, actionable read.


Across both


  • Reporting & tooling: Build repeatable dashboards/scripts (not one-off notebooks) so these analyses run as an ongoing cadence, not a one-time project; present findings to product, ML, and infra stakeholders in a form they can act on.



What we're looking for


Must-have


  • 0–2 years of experience - new grad or early-career, in a data/product analyst, applied ML, or research-adjacent role. We're hiring for raw analytical strength and trainability, not a finished track record.
  • Strong SQL and Python (pandas at minimum) from coursework, internships, or prior work - comfortable writing and debugging your own queries and scripts without hand-holding.
  • Solid statistical fundamentals (distributions, basic hypothesis testing, agreement/reliability concepts) - you don't need production experience with inter-rater reliability metrics, but you should pick up a new stats concept quickly when pointed at it.
  • Genuine comfort with ambiguous, messy real-world data - able to notice when something looks off and flag it clearly, even if the resolution isn't yours to call.
  • Basic funnel/cohort analysis instincts - comfortable thinking in terms of drop-off stages and segments, even if you haven't formally run a growth funnel before.
  • Native or near-native fluency in Hindi (or another Indian language) and English, with comfort reading/labeling code-mixed (Hinglish) text.


Strong plus


  • Any exposure to LLM evaluation concepts (prompt-based scoring, LLM-as-judge), speech/ASR evaluation (WER, transcript QA), or eval frameworks like DeepEval - even coursework or personal projects count.
  • Any exposure to product/growth analytics - funnel analysis, retention curves, cohort behavior - from a prior role, internship, or personal project.
  • Familiarity with cloud inference economics (e.g. token-based billing, provisioned throughput models) is a bonus, not expected.
  • A portfolio of self-directed analysis (a project, competition, or writeup) that shows you go looking for the "so what," not just the number.



First 90 days - success looks like


  • Running the LLM benchmarking comparison for post-call extraction end-to-end under direction — executing the sweep, producing clean cost/accuracy/fill-rate tables - with judgment calls escalated rather than made solo.
  • Contributing meaningfully to the golden dataset build: flagging inconsistencies in transliteration/script conventions clearly enough that a fast decision can be made, then applying that decision consistently across the dataset.
  • Running the inter-rater reliability pipeline on a recurring basis once it's set up, without needing to redesign it each time.
  • Has produced a clear, first-pass view of the self-serve funnel (signup → activation → habit → expansion) with at least one concrete drop-off point identified and flagged for the team to act on.
  • Has become the reliable first pass on "pull me the data on X" for at least two of: model benchmarking, ASR evaluation, routing/latency, agent behavior analytics, self-serve/usage patterns - freeing senior time for interpretation rather than execution.



What we offer


  • Innovative culture: You’ll be part of a generational opportunity to define the trajectory of AI, surrounded by a team pushing the boundaries of what’s possible.
  • Growth paths: Joining Bolna means joining a dynamic team with countless opportunities to drive impact - beyond your immediate role and responsibilities.
  • Learning & development: Bolna proactively supports professional development, the processes for which are being set up.
  • Competitive compensation + meaningful ESOP
  • Bengaluru office, in-person team collaboration

Demand for Analyst roles

New Analyst postings at the employers we follow are up 26% over the last 6 weeks compared with the 6 before.

Weekly new Analyst postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 993, the most 1,510, and the latest week had 1,367.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under Analyst, by the work the posting describes.

Analysts answer business questions with data. They pull numbers with SQL, build reports and dashboards, and explain what changed and why to the people making decisions.

Most postings ask for SQL, spreadsheets, a dashboard tool such as Tableau or Power BI, and clear writing.

Unlike a data scientist, an analyst mostly explains what happened rather than building models that predict what will.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Bengaluru, India (Onsite)

Working pattern and location restrictions need checking in the full posting.

Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.