Quality Check

Once the annotators complete some of the jobs in GT create and these jobs reach the end step, you can perform a final quality check on a random sample from a batch.

‌To generate a QC sample, you first need to create a submission.

Generate a QC

1. Create Submissions

You can create a submission from the Final Stage tab in GT-manage. It shows a list of all the completed jobs grouped by date.

2. Generate a QC Task

a) Sample size: Percentage of jobs to sample from the batch

b) Frame Intervals: It is only applicable for Video projects. For longer sequences, we suggest keeping high frame intervals and a high sample percentage to ensure the sample represents many sequences.

c) Reviewers: Assign reviewers who will be performing the QC

After generating the sample, you can perform the QC task in GT create. To know how to perform a QC task, you can have a look here:

• QC for Image Segmentation

• QC for 2D Video & Image Labeling

QC Results

As the sample tasks get solved, the result updates in the Latest QC section.

1. QC Metrics

Segmentation

$$Class\ Precision = \frac{Total\ images\ with\ zero \ critical\ class\ mislabelled }{Total\ Images}$$

$$Geomtric\ Precision = \frac{Total\ images\ with\ zero\ critical\ geomety\ mistakes}{Total\ Images}$$

$$Instance\ Precision = \frac{Total\ images\ with\ zero \ instance\ precision\ mistakes }{Total\ Images}$$

$$Recall = \frac{Total\ images\ with\ zero\ critical\ missed\ object}{Total\ Images}$$

Video and Image

$$Precision = \frac{\sum_{t}TP_t}{\sum_{t}(TP_t+FP_t)}$$

$$Recall = \frac{\sum_{t}TP_t}{\sum_{t}(TP_t+FN_t)}$$

True Positive: If class, shape, tracker-ID, and all the attributes are correct

False Positve: If any of the above is incorrect for an annotation

False Negative: Annotation missed

🧠Note: In video annotation, an object in one frame counts as one annotation. For example, a car present in ten frames counts as ten annotations.

2. QC Details

To improve the quality of the data, you can analyze the mistakes made by your annotators. Click on View QC details to check:

• Class-Level accuracy

• Mistake-Type distribution

• Time spent in the QC by the Reviewers