Data Quality Specialist – AI/ML / Data Annotation

Job Requirements

• 2+ years of experience in data quality management, data operations, or related roles within AI/ML or data annotation environments.
• Experienced in data annotation processes, quality assurance methodologies, and statistical quality metrics (e.g., F1 score, inter-annotator agreement).
• Proficiency with annotation and QA tools (e.g., Labelbox, Dataloop, LabelStudio).
• Familiarity with the core concepts of AI/ML pipelines, including data preparation, model training, and evaluation.

Responsibilities

Data Analysis

• Quality Audits: Perform quality audits on annotated datasets to ensure that they meet established guidelines and quality benchmarks.
• Statistical Reporting: Leverage statistical-based quality metrics such as F1 score and inter-annotator agreement to evaluate data quality.
• Root Cause Analysis: Analyze annotation errors, trends, project processes, and project documentation to identify and understand the root cause of errors and propose remediation strategies.
• Edge-Case Management: Resolve and analyze edge-case annotations to ensure quality and identify areas for improvement.
• Tooling: Become proficient in using annotation and quality control tools to perform reviews and track quality metrics.
• Guidelines: Become an expert in the project-specific guidelines and provide feedback for potential clarifications or improvements.

Continuous Improvement

• Automation: Identify opportunities to use automation to enhance analytics, provide deeper insights, and improve efficiency.
• Documentation: Develop and maintain up-to-date documentation on quality standards, annotation guidelines, and quality control procedures.
• Feedback: Provide regular feedback that identifies areas for improvement across the annotation pipeline.

Collaboration & Communication

• Cross-Functional Teamwork: Work closely with key project stakeholders and clients to understand project requirements and improve annotation pipelines.
• Training: Assist with training annotators, providing guidance, feedback, and support to ensure data quality.
• Reporting: Provide regular updates that highlight data quality metrics, key findings, and actionable insights for continuous process improvements.