AI Agentic Workflow Reviewer
Innodata Inc.
$50 - $55 / hour
Need a reasonable accommodation to apply or interview? Contact us.
JobMinglr uses automated technology to recommend jobs based on profile information and job preferences. Match Score does not determine eligibility for a position, prevent a user from viewing or applying to a job, or make hiring decisions on behalf of an employer.
Description
Innodata (Nasdaq: INOD) is a global data engineering company. We believe that data and Artificial Intelligence (AI) are inextricably linked. Our mission is to enable the responsible advancement of artificial intelligence by providing the data, evaluation frameworks, and human expertise required to build AI systems that can be trusted at scale. We provide a range of transferable solutions, platforms, and services for Generative AI / AI builders and adopters. In every relationship, we honor our 36+ year legacy delivering the highest quality data and outstanding outcomes for our customers.
Scope of the Role:
We're building a dedicated reviewer team to evaluate complex, real-world agentic AI workflows, assessing whether AI agents complete tasks safely, respect user intent and consent, and hold up under close scrutiny. This isn't routine content moderation: reviewers work through ambiguous, multi-step scenarios where judgment calls matter as much as checklists. You'll operate inside isolated test environments, apply structured rubrics, and help calibrate what "safe and policy adherent agent behavior" actually looks like.
What You’ll Own:
- Review multi-step agent task trajectories against rubric-based criteria covering task success, safety, and policy adherence
- Evaluate whether an agent respected user agency and informed consent — not just whether it followed literal instructions
- Apply privacy guardrails when reviewing tasks involving sensitive or personal data
- Work inside isolated test environments, including verifying environment resets between test runs to prevent cross-contamination
- Flag and escalate safety-relevant or ambiguous findings through defined escalation paths
You’ll Thrive in This Role If You Have:
- Bachelor's degree or equivalent practical experience
- 2–5+ years in quality review, QA/QC, trust & safety, content moderation, data annotation, or a similarly judgment-intensive review role
- Strong written communication – you can justify a scoring decision clearly enough for someone else to audit it
- Comfort with ambiguity and structured decision-making under a rubric, rather than needing a fixed rulebook for every case
- Baseline understanding of AI/ML concepts and how AI agents complete tasks (can be learned through onboarding, but some familiarity helps)
The expected hourly salary range for this position is $50-$55p/hour, based on experience, skills, and qualifications.
Please be aware of recruitment scams involving individuals or organizations falsely claiming to represent employers. Innodata will never ask for payment, banking details, or sensitive personal information during the application process. To learn more on how to recognize job scams, please visit the Federal Trade Commission’s guide at https://consumer.ftc.gov/articles/job-scams.
If you believe you’ve been targeted by a recruitment scam, please report it to Innodata at verifyjoboffer@innodata.com and consider reporting it to the FTC at ReportFraud.ftc.gov.
About this role
Innodata is looking for reviewers to evaluate AI agent workflows in controlled test environments, assessing whether autonomous systems complete tasks safely, respect user intent, and comply with policy. This is specialized quality work that goes beyond standard content moderation—you'll apply structured rubrics to ambiguous, multi-step scenarios where judgment and clear reasoning matter as much as adherence to criteria. The role involves flagging safety concerns, escalating edge cases, and helping define what responsible agent behavior looks like at scale.
You'll need 2–5+ years in quality assurance, trust and safety, content moderation, data annotation, or similar judgment-intensive review work. A bachelor's degree or equivalent experience is required. Strong written communication is essential—you need to justify your scoring decisions clearly enough that others can audit your work. The role suits someone comfortable making nuanced calls within a rubric rather than following a fixed rulebook, with at least baseline familiarity with AI and ML concepts (though this can be developed during onboarding).
This is a full-time, in-office position in San Jose at $50–$55 per hour based on experience and qualifications.
This role's local market on JobMinglr
Pay for this role
$50 to $55 per hour
That is the employer's posted range, and where you land in it is usually decided in one conversation. Before that conversation, run the numbers: analyze an offer for this role.
Before you apply, worth reading
How JobMinglr reads this job
Every listing here is scored against your profile before you apply: skills overlap, experience level, location and work arrangement, each weighted and explained. You see the score and the reasons, not just a list. How the matching works.