QC Tool Overview - Image & Video Annotation
Last updated
Last updated
The Maker, Reviewer, and QC Tools share some common components and features along with their own unique ones. Below are the ones that are common across all three:
It is the space in which the image that needs labeling is placed.
There are multiple features to adjust the size, position, and quality of the image that can help you annotate accurately or check the created annotations with precision.
For a maker tool, the header has tools that have various Annotating methods.
For a QC tool, the header has tools to Mark Issue and measure objects using Ruler.
For Reviewer, the header will display the options to either Edit or Reject the job.
Show/ Hide Issues: By default, the 'Issues' checkbox is selected. It can be checked to either show or hide the issues present in the frame/frames.
Show/ Hide Labels: Labels of the annotations created can be seen by hovering the cursor upon them. However, to have all labels visible by default, the 'Label' checkbox can be checked.
Show/ Hide Annotations: On checking the 'Annotation' checkbox, all the annotations present in canvas get hidden. This is done to have a better understanding of the background at times
Check if annotations and their corresponding attributes are marked correctly across frames in a glimpse by viewing them under different groups.
Navigate across different frames of the video sequence or jump to a particular frame using the seek bar.
When the tool loads, the annotation list on the right sidebar indicates the status of correction for every listed annotation. By default, the status always indicates as incomplete in the form of a 'question mark'.
Upon selecting an annotation, the annotation detail panel slides open. There are selectable crosses and ticks for every corresponding detail. Based on your actions, you may encounter Below is the list of statuses you may encounter based on the annotation
a) Incomplete
When all details are left unmarked.
b) Correct
When all details are marked as correct.
c) Incorrect
This can appear in two cases:
When all details are marked as incorrect.
When one or few details are marked as incorrect and the rest, as correct.
d) Partially Correct
When one or few details are marked as correct and the rest, left unmarked.
e) Partially Incorrect
This can appear in two cases:
When one or few details are marked as incorrect and the rest, unmarked.
When one or few details are marked as incorrect, some as correct, and some left unmarked.
Upon selecting an annotation, the annotation detail panel slides open. There are crosses and ticks for every corresponding detail that can be selected.
On clicking 'Mark unmarked as correct below' all the details that are incomplete, get marked as correct. The status of the annotation on the Annotation list will then reflect appropriately.
On clicking 'Mark unmarked as correct - across frames' over the Annotation List, all incomplete details for annotations across frames gets marked as corrected.
Tips for Efficiency :
Use the View by Group feature to easily check the class and attributes in a frame and across frames for an annotation
Mark what is incorrect in every frame and then click 'Mark unmarked as correct - across frames' button over the Annotation list right at the end before submitting. This will mark everything else that is incomplete as correct with just one click.
The description is a tracker-level description
If you see any error in the existing annotation, you can add the description by selecting the annotation
Definitions
Issue - Issue is any type of quality error observed while doing the quality check. It could be of 3 types -
Missed annotation - If an annotation is missing for an object in the image/frame.
Annotation level - If there's an issue with an existing annotation like geometry, class, etc.
Other - If you encounter an issue that doesn't belong to the above categories e.g. a guideline change, or a misc case.
Description - Each issue has its own description where you can add more details about the issue.
For missed annotation and annotation level issues it is optional.
For other issues adding a description is mandatory.
For Missed Annotation, you'll see a Mark Issue button on the top left, clicking on this will enable drawing a doodle which you can use to mark the location in the frame/image where you see the error.
Once you have added an issue using doodle, it'll be selected as missed annotation by default. You can change it according to the type of issue you observed.
For Missed Annotation issue
Class it belongs to - This is the class of the missing object. (Mandatory)
No. of missed objects - It is used to depict the number of missed objects in that frame (For video tasks, the counter will get reset in each frame)
Description - An optional field to add more details about the issue. You can edit/delete the description before submitting the task.
No. of missed objects is used to calculate the recall metric for the QC task.
For Other issues, you'll see a Mark Issue button on the top left, clicking on this will enable drawing a doodle which you can use to mark the location in the frame/image where you see the error
For Other Issue
Description - It will be mandatory to add a description.
Other issues won't be used for QC metric calculation, it's purpose is to get subjective feedback for any miscellaneous case
Here's a demo video explaining the various examples of issues and how to mark them.
