QC Tool Overview - Image & Video Annotation

Components of the Tools

‌The Maker, Reviewer, and QC Tools share some common components and features along with their own unique ones. Below are the ones that are common across all three:‌

1. Canvas

It is the space in which the image that needs labeling is placed.

2. Image adjustment controls

There are multiple features to adjust the size, position, and quality of the image that can help you annotate accurately or check the created annotations with precision.

3. Header Tools

For a maker tool, the header has tools that have various Annotating methods.
For a QC tool, the header has tools to Mark Issue and measure objects using Ruler.
For Reviewer, the header will display the options to either Edit or Reject the job.

4. Annotation Visibility Controls

Show/ Hide Issues: By default, the 'Issues' checkbox is selected. It can be checked to either show or hide the issues present in the frame/frames.
Show/ Hide Labels: Labels of the annotations created can be seen by hovering the cursor upon them. However, to have all labels visible by default, the 'Label' checkbox can be checked.
Show/ Hide Annotations: On checking the 'Annotation' checkbox, all the annotations present in canvas get hidden. This is done to have a better understanding of the background at times

5. View by Group

Check if annotations and their corresponding attributes are marked correctly across frames in a glimpse by viewing them under different groups.

6. Seek bar (for sequential images)

Navigate across different frames of the video sequence or jump to a particular frame using the seek bar.

Performing a QC

Existing Annotations

1. Annotation List

‌When the tool loads, the annotation list on the right sidebar indicates the status of correction for every listed annotation. By default, the status always indicates as incomplete in the form of a 'question mark'.‌

2. QC Status

‌Upon selecting an annotation, the annotation detail panel slides open. There are selectable crosses and ticks for every corresponding detail. Based on your actions, you may encounter Below is the list of statuses you may encounter based on the annotation

a) Incomplete

When all details are left unmarked.

b) Correct

When all details are marked as correct.

c) Incorrect

This can appear in two cases:

When all details are marked as incorrect.
When one or few details are marked as incorrect and the rest, as correct.

d) Partially Correct

When one or few details are marked as correct and the rest, left unmarked.

e) Partially Incorrect

This can appear in two cases:

When one or few details are marked as incorrect and the rest, unmarked.
When one or few details are marked as incorrect, some as correct, and some left unmarked.

3. How to conduct QC?

Upon selecting an annotation, the annotation detail panel slides open. There are crosses and ticks for every corresponding detail that can be selected.
On clicking 'Mark unmarked as correct below' all the details that are incomplete, get marked as correct. The status of the annotation on the Annotation list will then reflect appropriately.

On clicking 'Mark unmarked as correct - across frames' over the Annotation List, all incomplete details for annotations across frames gets marked as corrected.

Tips for Efficiency :

Use the View by Group feature to easily check the class and attributes in a frame and across frames for an annotation
Mark what is incorrect in every frame and then click 'Mark unmarked as correct - across frames' button over the Annotation list right at the end before submitting. This will mark everything else that is incomplete as correct with just one click.
The description is a tracker-level description

4. Adding Description

If you see any error in the existing annotation, you can add the description by selecting the annotation

Marking an Issue

Definitions

Issue - Issue is any type of quality error observed while doing the quality check. It could be of 3 types -
- Missed annotation - If an annotation is missing for an object in the image/frame.
- Annotation level - If there's an issue with an existing annotation like geometry, class, etc.
- Other - If you encounter an issue that doesn't belong to the above categories e.g. a guideline change, or a misc case.
Description - Each issue has its own description where you can add more details about the issue.
- For missed annotation and annotation level issues it is optional.
- For other issues adding a description is mandatory.

1. Missed Annotation

For Missed Annotation, you'll see a Mark Issue button on the top left, clicking on this will enable drawing a doodle which you can use to mark the location in the frame/image where you see the error.

Once you have added an issue using doodle, it'll be selected as missed annotation by default. You can change it according to the type of issue you observed.
For Missed Annotation issue
- Class it belongs to - This is the class of the missing object. (Mandatory)
- No. of missed objects - It is used to depict the number of missed objects in that frame (For video tasks, the counter will get reset in each frame)
- Description - An optional field to add more details about the issue. You can edit/delete the description before submitting the task.

No. of missed objects is used to calculate the recall metric for the QC task.

2. Other Issue

For Other issues, you'll see a Mark Issue button on the top left, clicking on this will enable drawing a doodle which you can use to mark the location in the frame/image where you see the error
For Other Issue
- Description - It will be mandatory to add a description.