Job Details

Operations Eng/Annotation Lead

Position Details:

Location: San Francisco, CA 94107 (Onsite)

Type: Contract

$40-50w2

1. Communicate with vendors
2. Possibly build dashboard
3. Communication between engineers and annotation team internally and externally. SO the accuracy of the annotation is important.

Responsibilities

1. Annotation Lead

Own the end-to-end quality and coordination of annotation workflows across concurrent streams.

• Ensure annotation outputs consistently meet the quality bar; identify and resolve systematic error patterns (label inconsistencies, missed edge cases, boundary errors)
• Serve as the primary interface between ML engineers and the external annotation vendor — ensuring timely and crisp communication on annotation tasks including clarifications, guideline alignment, and QA feedback
• Work with engineers to establish and maintain annotation guidelines, protocols, and gold-set standards; verify ground truth sets against engineer intent before large-scale annotation begins
• Identify and implement pre-annotation tooling or workflow improvements that increase annotator efficiency and reduce error rates
• Track inter-annotator agreement, QA pass rates, and rejection patterns over time; use findings to drive continuous guideline improvement

2. System Performance Monitoring

Develop a deep understanding of where and how the system struggles, and keep engineering informed.

• Build intuition on hard corner cases and edge conditions where the system fails — through ongoing review of annotation outputs, rejection patterns, and model validation results
• Communicate regularly with engineering to surface failure patterns, flag emerging issues, and play an active role in driving identified issues to resolution
• Track pipeline health metrics by building and maintaining whatever tooling is appropriate (SQL queries, scripts, spreadsheets, or lightweight dashboards)

What We're Looking For

• Able to pick up and use tools such as SQL, Python, dashboard platform to pull, summarize, and track metrics.
• Strong analytical eye — able to spot quality issues and failure patterns in structured label data
• Clear, crisp communicator; comfortable as the connective tissue between engineering and an external vendor
• Organized and self-directed; able to maintain tracking systems and keep multiple workstreams moving without close oversight
• Work in the US business hours

Requirements:

• A Bachelor's Degree with 3-4 years of experience or a Master's Degree with 2 years' experience in Computer Science, Electrical Engineering, or a related field.
• Core Skills: General Software Engineering skills with 3+ years of programming experience in python and the surrounding tooling ecosystem along with familiarity in linux and expertise in infrastructure, cloud and/or MLOps,.
• Personal Attributes: Team player, good communication skills, self starter.
• Strong teamwork and communication skills to collaborate with cross-functional teams, including ML and software engineers.
• Nice to Have: Experience building MLOps pipelines for deep learning based perception solutions on AWS or GCP

EEO Statement:

Blackstone Talent Group is a division of Blackstone Technology Group, a global IT services and solutions firm that implements technological solutions across commercial industry verticals and the US Federal Government. Blackstone's global talent augmentation practice was founded in 1998. Blackstone Talent Group has offices in San Francisco, Denver, Houston, Colorado Springs, and Washington, DC. We specialize in providing clients the best talent across a variety of industries and sectors.