QC Tool Overview - 2D-3D Linking

Components of the tools

Tool's basic functions

Bottom Bar

Multi-camera Image Support

View by Group

Navigation

Performing a QC

Existing Annotations

1. Annotation List

‌When the tool loads, the annotation list on the right sidebar indicates the status of correction for every listed annotation. By default, the status always indicates as incomplete in the form of a 'question mark'.‌

2. QC Status

‌Upon selecting an annotation, the annotation detail panel slides open. There are selectable crosses and ticks for every corresponding detail. Based on your actions, you may encounter Below is the list of statuses you may encounter based on the annotation

a) Incomplete

When all details are left unmarked.

b) Correct

When all details are marked as correct.

c) Incorrect

This can appear in two cases:

• When all details are marked as incorrect.

• When one or a few details are marked as incorrect and the rest, as correct.

d) Partially Correct

When one or a few details are marked as correct and the rest, left unmarked.

e) Partially Incorrect

This can appear in two cases:

• When one or few details are marked as incorrect and the rest, unmarked.

• When one or few details are marked as incorrect, some as correct, and some left unmarked.

Here's the video showing all the status

3. How to conduct QC?

• Upon selecting an annotation, the annotation detail panel slides open. There are crosses and ticks for every corresponding detail that can be selected.

• On clicking 'Mark unmarked as correct below' all the details that are incomplete, get marked as correct. The status of the annotation on the Annotation list will then reflect appropriately.

• On clicking 'Mark unmarked as correct - across frames' over the Annotation List, all incomplete details for annotations across frames get marked as corrected.

Tips for Efficiency :

• Use the View by Group feature to easily check the class and attributes in a frame and across frames for an annotation

• Mark what is incorrect in every frame and then click the 'Mark unmarked as correct - across frames' button over the Annotation list right at the end before submitting. This will mark everything else that is incomplete as correct with just one click.

• The description is a tracker-level description

4. Adding Description

If you see any error in the existing annotation, you can add the description by selecting the annotation

Marking an Issue

Definitions

• Issue - Issue is any type of quality error observed in the task. It could be of 3 types -

  • Missed annotation - If an annotation is missing.

  • Annotation level - If something is wrong with an existing annotation like geometry, class, attribute, etc.

  • Other - If any error doesn't belong to the above categories e.g. a guideline change, or a misc case.

• Description - Each issue will have its own description which contains the details about the issue.

  • For missed annotation and annotation level issues it is optional.

  • For other issues adding a description is mandatory.

How to mark Missed annotations

1. For Missed and Other issues you'll see a Mark Issue button on the top left,

   If the primary view is LiDAR

   • Clicking on the Mark issue button will enable drawing a beacon which you can use to mark the location in the LiDAR view where you see the error.

   • The beacon can be dropped only on a point, if you observe that dropping a beacon is not working, then it’s because it’s not on a point.

If the primary view is Camera

• Clicking on the Mark issue button will enable drawing a doodle which you can use to mark the location in the frame/image where you see the error

  • You will have to change the camera image in the primary view to mark errors in other cameras in the same frame

2. Once you have added an issue using a doodle or beacon, it'll be selected as missed annotation by default. You can change it according to the type of issue you observed.

• For Missed annotation Issue -

  • Adding a class is mandatory

  • The counter - No. of missed objects is used to depict the number of missed objects in that frame.

    This will be used to calculate the recall metric for the QC task.

  • Description - It is optional to add the description for missed annotation issues.

• For Other issue -

  • Description - It will be mandatory to add a description.

Other issues won't be used for QC metric calculation, its purpose is to get subjective feedback for any misc case.

• If you see any error in the existing annotation, you can add the description by selecting the annotation.

  • You’ll be able to add separate descriptions for each sensor where the object is present

The description is a tracker-level description.

3. Issues in the right-side panel will be active only for the primary sensor.

• In the below image LiDAR view is the primary view hence the camera issues are disabled.

• When you switch to the camera view, the issue present in that particular camera sensor will become active/clickable and the rest of them will become inactive.