About us Coalition is the world's first Active Insurance provider designed to help prevent digital risk before it strikes. Founded in 2017, Coalition combines comprehensive insurance coverage and innovative cybersecurity tools to help businesses manage and mitigate potential cyberattacks. Opportunities to make an impact with bold thinking are real—and happening daily at Coalition.

About The Role

Coalition's machine learning models are only as good as the data they're trained on. This role exists to make sure that data is right. You'll own labeling quality and methodology across Coalition — designing annotation tasks, defining quality frameworks, and ensuring every labeled dataset meets the standard required to ship production ML models.

You'll manage our labeling platform (Label Studio), work directly with ML and product teams to structure labeling programs, and oversee outsourced labeling vendors to hit quality and throughput targets. This role reports to the Chief Product Officer and sits at the intersection of product, ML, and operations. You won't manage internal labelers — all annotation work is outsourced — but you will be the single point of accountability for whether Coalition's labeled data is accurate, consistent, and fit for purpose.

Responsibilities

Labeling quality methodology : Define annotation guidelines, taxonomies, and edge-case protocols for each labeling program. Establish gold standard datasets, inter-annotator agreement (IAA) targets, and audit sampling processes. Identify and remediate mislabeled data in existing datasets.

Requirements

driver for Label Studio — defining project configuration needs, workflow designs, pre-labeling pipeline

, and integration points with ML infrastructure. Partner with the data engineering team that builds and maintains the platform.

into well-structured labeling tasks. Challenge teams on task design when labeling instructions are ambiguous or likely to produce unreliable labels. Vendor management : Source, onboard, and manage external labeling vendors and BPOs in coordination with Coalition's operations team.

Set quality SLAs, run calibration sessions, and manage feedback loops to labelers. Hold vendors accountable to accuracy, not just throughput. Measurement improvement : Define and track operational metrics — label accuracy, IAA scores, cost per label, turnaround time — and use them to drive continuous improvement.

Identify opportunities for active learning, model-assisted labeling, and pre-annotation to reduce cost without sacrificing quality.

Qualifications

5+ years in ML data operations, data labeling, or a related field (ML engineering, data science, or data engineering with heavy labeling exposure) Deep understanding of annotation quality frameworks: IAA, consensus labeling, gold standard evaluation, error taxonomy, and calibration workflows Direct experience managing labeling platforms (Label Studio strongly preferred; Scale AI, Labelbox, Prodigy, or similar acceptable) Track record managing outsourced labeling vendors or BPOs for ML data production Familiarity with common ML labeling tasks: text classification, NER, document extraction, intent detection Comfortable working in Python and SQL; bonus if you've built tooling around labeling workflows or quality measurement Strong opinions on what makes labeled data good or bad, and the willingness to push back when it's bad Experience in insurance, cybersecurity, or fintech is a plus but not required

Compensation

As a remote-first organization, our

reflects the cost of labor across several Canadian geographic markets. In Alberta, British Columbia Ontario the base salary for this position ranges from $136,800/year up to $186,800/year.

Senior Manager, Machine Learning (Data Operations)