Scene 1

Automatic labeling of vulnerable road users in multi-sensor data

Martin Dimitrievski , Ivana  Shopovska , David Van Hamme, Peter  Veelaert , Wilfried Philips IMEC-IPI- Ghent  University Ghent , Belgium

The 24th IEEE International Conference on Intelligent Transportation Systems September 19 – 22, 2021 Virtual Conference

Scene 2

Introduction Contributions Dataset  details and labeling Evaluation results Discussion

Scene 3

Perception in autonomous vehicles is driven by deep learning. There exist many datasets that enable training and testing of perception models.  ( Waymo ,  nuScenes ,  Level5, BDD, A2D2, KITTI ,  Argoverse , Oxford  Robotcar , etc.) Issues: Limited variability  of traffic and weather  conditions Lack track-level annotations Lack key sensors such as  radar or thermal Lack metadata

Scene 4

Many  companies and institutes perform “in-house” data labeling . Labeling automation is often applied in order to reduce the cost and effort. In this paper we present an  automatic people labeling  method for multi-sensor data.  Track level annotations Detailed analysis of the ground truth quality

Scene 5

Manual labeling Semi-automatic labeling Fully  a utomatic labeling Annotation  time ●●●● ●● ● Cost ●●● ●● ● Effort and fatigue ●●●● ●● ● Quality  Excellent Very good Unknown

Scene 6

Scene 7

Method  for automatic and semi-automatic 2.5-D data  labeling. Detailed quality assessment of the computed ground  truth. A novel, high quality 2.5-D people tracking dataset containing raw radar  signals. Dataset details: 316 sequences  (10s-20s),  81K frames ,  1318 unique VRUs ,  173K  instances Additional  metadata:  light conditions, occlusion level, VRU sub-class, VRU height, group forming behavior and ego-motion.

Scene 8

Scene 9

Scene 10

Scene 11

Scene 12

Input
Per-frame
processing:
Intermediate outpu
Image sequence
Object
detection
Human annotator
Add, remove, adjust
Data fusion
Boun Ing
boxes
Depth
image
Point cloud sequence
Depth image
calculation
Temporal
processing:
Output
Optical flow
calculation
BB flow
vectors
detections:;
Forward pass (Kalman filter)
association: Munkres 2D-3D algorithm;
prediction: constant velocity model;
updates: every 5th frame with intermediate output,
every frame with optical flow vectors
Backward pass
Re-activate parts of trajectories with weak confidence
tracks:;
Ego-motion
Odometry
estimation
Off-line correction

Scene 13

Scene 14

How accurate is the proposed automatic labeling under various conditions? Experiment: Automatically computed labels are tested against manually corrected labels. Data is split into sub-sets  according  to various  criteria. Metrics: AP, F-2 score, MOTA, MOTP, MT/PT/ML, IDS, FRAG. Goals: Assess the qualit y of automatic labeling in a general case. Identify conditions  where  automatic  labeling  performance  is unreliable.

Scene 15

Evaluation criteria: Light  conditions 		(Daytime , Twilight,  Nighttime) Occlusion level 		(Visible , partly occluded,  occluded) VRU sub-category  ( pedestrians vs. cyclists) VRU  height Group forming behavior EGO  velocity 				More details in  the paper.

Scene 16

Proposed  a utomatic labeling MOTA ↑ 0.647 MOTP [m]↓ 0.277 AP ↑ 0.833 F-2 score  ↑ 0.836 MT [%] ↑ 69.8 TID [ ms ]↓ 232 IDS [%]↓ 0.45 FRAG [%]↓ 0.81

Manual labeling Semi-automatic labeling Automatic labeling Scene complexity ∗ 20s 1.8s 1.2s † 40s 4.0s 1.2s

Scene 17

AP↑ Dataset average Day Twilight Night 0.83299999999999996 0.86799999999999999 0.83699999999999997 0.72199999999999998

MOTA↑ Dataset average Day Twilight Night 0.64700000000000002 0.70199999999999996 0.65500000000000003 0.47099999999999997

Scene 18

AP↑ Dataset average Visible Partly occluded Occluded 0.83299999999999996 0.871 0.81799999999999995 0.70499999999999996

MOTA↑ Dataset average Visible Partly occluded Occluded 0.64700000000000002 0.70499999999999996 0.622 0.46500000000000002

Scene 19

AP↑ Dataset average Mostly short Average height Mostly tall 0.83299999999999996 0.75600000000000001 0.85299999999999998 0.92400000000000004

MOTA↑ Dataset average Mostly short Average height Mostly tall 0.64700000000000002 0.60099999999999998 0.66100000000000003 0.66100000000000003

Scene 20

AP↑ Dataset average Mostly pedestrians Pedestrians and cyclists Mostly cyclists 0.83299999999999996 0.83899999999999997 0.84799999999999998 0.82299999999999995

MOTA↑ Dataset average Mostly pedestrians Pedestrians and Cyclists Mostly cyclists 0.64700000000000002 0.68500000000000005 0.623 0.628

Scene 21

AP↑ Dataset average Static Moving 0.83299999999999996 0.85599999999999998 0.82499999999999996

MOTA↑ Dataset average Static Moving 0.64700000000000002 0.65700000000000003 0.64300000000000002

Scene 22

AP↑ Dataset average No groups Contains groups 0.83299999999999996 0.83599999999999997 0.82099999999999995

MOTA↑ Dataset average No groups Contains groups 0.64700000000000002 0.63600000000000001 0.69699999999999995

Scene 23

Aautomated   labeling is more effective when the camera object detector is operating in good light conditions, and the fully automatic annotations are reliable. However, the quality of the labels for “nighttime” are significantly lower than the average. This is especially evident for the  tracking  accuracy (MOTA), fraction of tracked trajectories (MT) and average precision (AP), which show marked decrease, while the fraction of lost tracks (ML) and the tracking delay (TID) are significantly higher. We conclude that in night time sequences, a manual intervention is often warranted.

Another major factor in the quality of automatic annotation is the level of occlusion. All tracking and detection metrics for the “visible” category are significantly above the dataset average, while in the case of “partly occluded” we observe below average quality across all metrics. The automatically computed labels in the “occluded” category suffer the highest loss of quality among all evaluated criteria and it is in these cases that human intervention is most needed.

In the case of labeling quality of different VRU subcategories, there is a slight advantage for the category of pedestrians versus cyclists. We deem that the annotation quality is almost equally good in both cases and these labels can be used for supervised training.

In the case of VRU height, our automatic labeling method gives an advantage for the category “mostly tall”. This is especially prominent in the detection metric AP, with more than 10% above the dataset average. Unfortunately, the automatically computed labels for the category “mostly short” suffer from a significant loss of quality as measured by all tracking and detection metrics. This finding is not surprising, since shorter objects cover a smaller image area, affecting the accuracy of detection and classification. We note that the split based on height is also dependent on the demographic distribution.

With regard to ego-motion, there is a slight drop in annotation quality when the vehicle is moving, as opposed to when it is static. However, tracking and detection performance are sufficient for both cases and no systematic additional human intervention is warranted for faster ego-motion scenarios.

In the analysis of group forming behavior we observed mixed results. The quality of the automatic labels measured by MOTA is slightly higher when the sequence contains groups, while the detection performance measured by AP is slightly lower, which is counter intuitive. We observe the same inconclusive pattern when analyzing the proportion of mostly tracked, partly tracked and mostly lost trajectories. Possibly the parameters which define a group, which were adopted from [22], need to be further tuned to the specific traffic situations, or we need to further differentiate between group layouts e.g. longitudinal queues or pairs of pedestrians walking side by side. The group forming behavior warrants further, more detailed, analysis. We conclude this section with an estimate of the gains in human annotation effort when using the proposed method. Since measuring human effort is a difficult task depending on highly subjective inputs, we refrain from measuring complex effects such as loss of attention, annotator fatigue or annotator discrepancy. Instead, in Table II we present only the average time needed to label different sequences (expressed as time per frame) as a general metric of annotation effort. Based on these results, we conclude that the proposed pre-labeling followed by manual correction speeds up the annotation by a factor of 6-7x as compared to fully manual labeling. Moreover, during manual labeling, we encountered significant annotator fatigue setting in much faster than when using a pre- labelled  solution. This limiting factor reduces the total amount of sequences a person can label in a given work day and requires further investigation.

Scene 24

Auto-labels Dataset average Day Twilight Night Visible Partly occluded Occluded Mostly pedestrians Mixed Mostly cyclists Mostly short Mixed Mostly tall Static Moving No groups Contains groups 0.64700000000000002 0.70199999999999996 0.65500000000000003 0.47099999999999997 0.70499999999999996 0.622 0.46500000000000002 0.68500000000000005 0.623 0.628 0.60099999999999998 0.66100000000000003 0.66100000000000003 0.65700000000000003 0.64300000000000002 0.63600000000000001 0.69699999999999995

Scene 25

Multi-Object Tracking Precision  MOTP[m ]↓

Auto-labels Dataset average Day Twilight Night Visible Partly occluded Occluded Mostly pedestrians Mixed Mostly cyclists Mostly short Mixed Mostly tall Static Moving No groups Contains groups 0.27700000000000002 0.28399999999999997 0.26400000000000001 0.29399999999999998 0.25700000000000001 0.28699999999999998 0.34200000000000003 0.27400000000000002 0.27400000000000002 0.28000000000000003 0.30599999999999999 0.27 0.251 0.22500000000000001 0.29899999999999999 0.27600000000000002 0.28199999999999997

Scene 26

Auto-labels Dataset average Day Twilight Night Visible Partly occluded Occluded Mostly pedestrians Mixed Mostly cyclists Mostly short Mixed Mostly tall Static Moving No groups Contains groups 0.83299999999999996 0.86799999999999999 0.83699999999999997 0.72199999999999998 0.871 0.81799999999999995 0.70499999999999996 0.83899999999999997 0.84799999999999998 0.82299999999999995 0.75600000000000001 0.85299999999999998 0.92400000000000004 0.85599999999999998 0.82499999999999996 0.83599999999999997 0.82099999999999995

Scene 27

Auto-labels Dataset average Day Twilight Night Visible Partly occluded Occluded Mostly pedestrians Mixed Mostly cyclists Mostly short Mixed Mostly tall Static Moving No groups Contains groups 0.83599999999999997 0.86699999999999999 0.83899999999999997 0.73699999999999999 0.86699999999999999 0.82299999999999995 0.72799999999999998 0.84399999999999997 0.84 0.82699999999999996 0.77800000000000002 0.85 0.89100000000000001 0.85099999999999998 0.82899999999999996 0.83599999999999997 0.83599999999999997

Scene 28

Proportion of mostly tracked objects MT[%] ↑

Auto-labels Dataset average Day Twilight Night Visible Partly occluded Occluded Mostly pedestrians Mixed Mostly cyclists Mostly short Mixed Mostly tall Static Moving No groups Contains groups 69.8 76.5 67.900000000000006 53.6 77 68.400000000000006 54.8 71.8 72.900000000000006 66.3 60.3 73.3 76.7 69.2 69.900000000000006 69.400000000000006 72.8

Scene 29

Auto-labels Dataset average Day Twilight Night Visible Partly occluded Occluded Mostly pedestrians Mixed Mostly cyclists Mostly short Mixed Mostly tall Static Moving No groups Contains groups 232 197 213 400 221 229 277 196 252 253 323 199 160 282 222 226 262

Scene 30

Auto-labels Dataset average Day Twilight Night Visible Partly occluded Occluded Mostly pedestrians Mixed Mostly cyclists Mostly short Mixed Mostly tall Static Moving No groups Contains groups 0.45 0.57999999999999996 0.36 0.34 0.41 0.48 0.46 0.47 0.56000000000000005 0.38 0.44 0.48 0.03 0.4 0.47 0.39 0.72

Scene 31

Auto-labels Dataset average Day Twilight Night Visible Partly occluded Occluded Mostly pedestrians Mixed Mostly cyclists Mostly short Mixed Mostly tall Static Moving No groups Contains groups 0.81 0.91 0.74 0.75 0.73 0.9 0.73 0.85 0.78 0.8 0.86 0.84 0.25 0.69 0.87 0.71 1.3

Scene 32

< Martin Dimitrievski > < PhD student > < IMEC-IPI Ghent university > E	Martin.Dimitrievski@ugent.be T	+32 9 264 34 12 M	+32 9 264 34 12 www.ugent.be

Ghent  University @ ugent Martin Dimitrievski

Automatic labeling of vulnerable road users in multi-sensor data

Scene 1 (0s)

Scene 2 (16s)

Scene 3 (24s)

Scene 4 (53s)

Scene 5 (1m 36s)

Scene 6 (2m 9s)

Scene 7 (3m 34s)

Scene 8 (3m 56s)

Scene 9 (4m 10s)

Scene 10 (4m 21s)

Scene 11 (4m 29s)

Scene 12 (4m 39s)

Scene 13 (5m 1s)

Scene 14 (5m 12s)

Scene 15 (5m 34s)

Scene 16 (5m 48s)

Scene 17 (6m 10s)

Scene 18 (6m 24s)

Scene 19 (6m 38s)

Scene 20 (6m 53s)

Scene 21 (7m 9s)

Scene 22 (7m 21s)

Scene 23 (7m 35s)

Scene 24 (9m 53s)

Scene 25 (10m 19s)

Scene 26 (10m 44s)

Scene 27 (11m 9s)

Scene 28 (11m 34s)

Scene 29 (11m 59s)

Scene 30 (12m 18s)

Scene 31 (12m 43s)

Scene 32 (13m 8s)