AI Data Analyst
Creative Tokyo
Salary undisclosed
Need a reasonable accommodation to apply or interview? Contact us.
JobMinglr uses automated technology to recommend jobs based on profile information and job preferences. Match Score does not determine eligibility for a position, prevent a user from viewing or applying to a job, or make hiring decisions on behalf of an employer.
Description
- Run ongoing human labeling for priority AI workflows, including response quality, task success, MCP and tool use, end-to-end Cowork-style workflows.
- Triage bad-query reports, downvotes, escalations, and routed quality issues; categorize failure modes and feed clean analysis back to partner teams.
- Perform qualitative analysis to identify recurring patterns such as hallucinations, retrieval failures, tool-use issues, weak grounding, and poor workflow completion.
- Follow and improve labeling guidelines and rubrics so judgments are consistent, realistic, and useful for model and product improvement.
- Maintain high-quality labeled datasets, golden sets, and regression slices; monitor coverage, drift, leakage, and difficulty distribution.
- Participate in calibration exercises with human graders and validate LLM-as-a-judge outputs against human labels to improve consistency and judge quality.
- Partner with engineers, PMs, QA, and eval owners on pre/post-change quality reads, benchmark updates, and release-readiness decisions.
- Contribute to recurring quality reporting by surfacing top failure modes, coverage gaps, quality shifts, and actionable recommendations.
- 3–5 years of experience in data labeling, data analysis, QA, or a related field.
- Strong analytical judgment and attention to detail, with the ability to apply nuanced rubrics consistently.
- Experience evaluating AI-generated outputs or working with NLP, search, recommendation, or other ML systems.
- Familiarity with structured qualitative analysis, basic SQL/data retrieval, and spreadsheet-based workflows.
- Clear written and verbal communication skills, with the ability to explain findings to both technical and non-technical audiences.
- Ability to work independently in ambiguous, fast-moving environments and collaborate effectively across functions.
By clicking “Submit Application,” I confirm that I have read the Global Data Privacy Notice and the Applicant Arbitration Agreement, and I agree to the terms.
Benefits
- stock options
About this role
Creative Tokyo is hiring an AI Data Analyst to build and maintain the human evaluation infrastructure that underpins Glean's AI product quality. You'd be the bridge between messy real-world quality issues and clean, actionable data—triaging user reports and escalations, labeling AI outputs against detailed rubrics, and identifying patterns like hallucinations, retrieval failures, and tool-use breakdowns. The work is deeply cross-functional: you'll partner with QA, engineering, product, and the evals team to convert ambiguous problems into structured signals that drive model and product improvements.
The role demands strong analytical judgment and consistency. You'll maintain labeled datasets and golden sets, run calibration exercises with other graders, validate LLM-as-a-judge outputs, and contribute to quality reporting that surfaces top failure modes and coverage gaps. This isn't a back-office task—your analysis directly informs release-readiness decisions and competitive benchmarking. You'll need 3–5 years in data labeling, QA, data analysis, or similar work; experience evaluating AI outputs or working with ML systems; and comfort with structured qualitative analysis, basic SQL, and spreadsheets.
The position is full-time, in-office four days a week in Bangalore. You should be comfortable working independently in ambiguous, fast-moving environments while communicating findings clearly to both technical and non-technical stakeholders. During the interview, expect a brief exercise on how you think about and use AI in your work.
This role's local market on JobMinglr
Pay for this role
The employer didn't post a pay range for this role. That usually means pay is set in negotiation, which favors whoever arrives with numbers. Check ranges on comparable AI Data Analyst postings in Bangalore, and analyze any offer before you accept it.
The path into this work
We map every route into data analyst work: what to learn, what each certificate costs, and how long each step takes. See the Data Analyst career path.
Before you apply, worth reading
How JobMinglr reads this job
Every listing here is scored against your profile before you apply: skills overlap, experience level, location and work arrangement, each weighted and explained. You see the score and the reasons, not just a list. How the matching works.