Evaluate segmentation and tracking results for the paper

========================================

"Joint Tracking and Segmentation of Multiple Targets". A. Milan, L. Leal-Taixé, K. Schindler and I. Reid. CVPR 2015

This package contains all necessary data and scripts to produce the numbers shown in Tables 2-4 of the paper. Results in Tab. 5 were produced using Yang Bo's evaluation script.

The baseline results for cl2, DP, DCO, and H2T are also included. cl2 and DP have been generated by us using the publicly available code, while DCO and H2T are the original results (bounding boxes), provided by the authors.

Note that the evaluation uses a slightly modified CLEAR_MOT.m where a minor bug has been fixed. The original implementation is also included as CLEAR_MOT_OLD.m but is not recommended to be used.

Contents

Prepare

clear all
addpath(genpath('.'));

Segmentation

fprintf('\n* * *   SEGMENTATION   * * *\n');

% load ground truth
load data/S2L2-gt-1-30
gtFrames = [1,5,10,15,30]; % these frames are annotated with masks
gt=gt(:,:,gtFrames);

% TSP
fprintf('\n--- Temporal Super-Pixels (TSP) ---\n');
load data/TSPSeg
evalSeg(gt, resTSP);

% Greedy labeling
fprintf('\n--- Greedy ---\n');
load data/GreedySeg
evalSeg(gt, resOursBaseline);

% our result
fprintf('\n--- Our result ---\n');
load data/ourSeg
evalSeg(gt, resOurs);

* * *   SEGMENTATION   * * *

--- Temporal Super-Pixels (TSP) ---
GT Regions: 	34
Res Regions: 	29

        CE |      PRCE |  over-seg |  ex. obj. |
      4.03 |     29.30 |      1.17 |      5.00 |

--- Greedy ---
GT Regions: 	34
Res Regions: 	29

        CE |      PRCE |  over-seg |  ex. obj. |
      4.13 |     25.63 |      1.17 |      7.00 |

--- Our result ---
GT Regions: 	34
Res Regions: 	24

        CE |      PRCE |  over-seg |  ex. obj. |
      3.56 |     24.34 |      1.42 |      7.00 |

Tracking

load ground truth

load data/gtInfos

fprintf('\n* * *   TRACKING   * * *\n');

* * *   TRACKING   * * *

Evaluation on ground plane on six sequences

fprintf('\n* * * * * * * * * * *\n');
fprintf('S2L1, S2L2, S2L3, S1L1-2, S1L2-1, TUD-Stadtmitte\n');
fprintf('Comparison to cl2, DP, DCO\n');
allscen=[23,25,27,71,72,42];  % S2L1, S2L2, S2L3, S1L1-2, S1L2-1, TUD-Stadtmitte
evopts.eval3d=1; evopts.td=1000;

fprintf('\n--- cl2 results...\n');
load data/cl2Infos
allmets=evalDataset(c2lInfos,gtInfos,allscen,evopts,0);

fprintf('\n--- DP results...\n');
load data/DPInfos
allmets=evalDataset(dpInfos,gtInfos,allscen,evopts,0);

fprintf('\n--- DCO results...\n');
load data/dcoInfos
allmets=evalDataset(dcoInfos,gtInfos,allscen,evopts,0);

fprintf('\n--- Our results...\n');
load data/ourInfos
allmets=evalDataset(infos,gtInfos,allscen,evopts,0);

* * * * * * * * * * *
S2L1, S2L2, S2L3, S1L1-2, S1L2-1, TUD-Stadtmitte
Comparison to cl2, DP, DCO

--- cl2 results...
 Rcll  Prcn   FAR| GT  MT  PT  ML|  FP   FN IDs  FM|  MOTA  MOTP MOTAL 
 72.2  79.9  1.06| 19   6  12   1| 846 1294  66  64|  52.6  72.8  53.9 
 28.3  81.0  1.56| 43   0  25  18| 682 7384 116 126|  20.5  68.6  21.6 
 28.8  82.0  1.15| 44   2  19  23| 276 3117  31  39|  21.8  68.9  22.4 
 40.6  77.8  1.85| 44   3  25  16| 447 2283  38  62|  28.0  68.4  29.0 
 28.7  96.8  0.24| 42   2  20  20|  48 3606  32  52|  27.1  59.5  27.7 
 48.2  45.4  3.75| 10   1   7   2| 671  599  16  91| -11.2  43.1 -10.0 
 ----------------------------------------------------------------------
 41.1  77.1  1.60| 34   2  18  13| 495 3047  50  72|  23.1  63.5  24.1 

--- DP results...
 Rcll  Prcn   FAR| GT  MT  PT  ML|  FP   FN IDs  FM|  MOTA  MOTP MOTAL 
 88.2  95.5  0.24| 19  17   2   0| 194  549 190 124|  79.9  74.6  84.0 
 38.6  95.4  0.44| 43   0  33  10| 193 6319 227 256|  34.5  61.3  36.7 
 38.3  94.7  0.39| 44   7  16  21|  93 2701 213 168|  31.3  64.9  36.1 
 61.4  96.9  0.31| 44  18  16  10|  75 1484 211 143|  54.0  66.0  59.4 
 25.4  97.1  0.19| 42   2  16  24|  38 3775 240 194|  19.9  64.2  24.6 
 68.7  71.3  1.79| 10   5   5   0| 320  362  96  44|  32.7  59.5  40.8 
 ----------------------------------------------------------------------
 53.4  91.8  0.56| 34   8  15  11| 152 2532 196 155|  42.1  65.1  46.9 

--- DCO results...
 Rcll  Prcn   FAR| GT  MT  PT  ML|  FP   FN IDs  FM|  MOTA  MOTP MOTAL 
 96.8  94.1  0.35| 19  18   1   0| 282  148  23  15|  90.3  74.2  90.7 
 65.1  92.4  1.26| 43  11  31   1| 549 3592 166 153|  58.2  59.8  59.7 
 43.2  94.6  0.45| 44   8  17  19| 108 2486  26  22|  40.1  65.3  40.7 
 64.9  93.7  0.70| 44  21  12  11| 169 1349  23  19|  59.9  61.9  60.5 
 30.9  98.3  0.13| 42   2  19  21|  27 3494  42  34|  29.6  58.8  30.4 
 69.1  85.6  0.75| 10   4   6   0| 134  357  15  13|  56.2  61.6  57.4 
 ----------------------------------------------------------------------
 61.7  93.1  0.61| 34  11  14   9| 212 1904  49  43|  55.7  63.6  56.6 

--- Our results...
 Rcll  Prcn   FAR| GT  MT  PT  ML|  FP   FN IDs  FM|  MOTA  MOTP MOTAL 
 97.5  89.8  0.65| 19  18   1   0| 517  117  28  17|  85.8  75.5  86.3 
 76.2  87.0  2.68| 43  22  21   0|1169 2448 161 156|  63.3  58.0  64.8 
 55.2  93.0  0.75| 44  13  17  14| 181 1962  44  40|  50.0  62.3  51.0 
 73.3  88.2  1.57| 44  22  15   7| 379 1026  30  26|  62.7  66.5  63.4 
 45.9  85.2  2.00| 42   9  19  14| 402 2736  56  53|  36.9  54.5  37.9 
 67.1  84.8  0.78| 10   3   7   0| 139  380   5   9|  54.7  62.9  55.0 
 ----------------------------------------------------------------------
 69.2  88.0  1.41| 34  15  13   6| 465 1445  54  50|  58.9  63.3  59.8

Evaluation on ground plane (cropped tracking area)

fprintf('\n* * * * * * * * * * *\n');
fprintf('\nComparison to CEM\n');
allscen=[23,25,27,71,72,42]; % S2L1, S2L2, S2L3, S1L1-2, S1L2-1, TUD-Stadtmitte

evopts.eval3d=1; evopts.td=1000;
fprintf('\n--- Our results...\n');
allmets=evalDataset(infos,gtInfos,allscen,evopts,1);


%
fprintf('\n* * * * * * * * * * *\n');
fprintf('Comparison to H2T...\n');
fprintf('S2L1, S2L2, S2L3, S1L1-1, S1L1-2\n');
load data/H2TInfos
allscen=[23,25,27,70,71]; % S2L1, S2L2, S2L3, S1L1-1, S1L1-2

evopts.eval3d=1; evopts.td=1000;

fprintf('\n--- H2T results...\n');
allmets=evalDataset(wenInfos,gtInfos,allscen,evopts,1);

fprintf('\n--- Our results...\n');
allmets=evalDataset(infos,gtInfos,allscen,evopts,1);

* * * * * * * * * * *

Comparison to CEM

--- Our results...
 Rcll  Prcn   FAR| GT  MT  PT  ML|  FP   FN IDs  FM|  MOTA  MOTP MOTAL 
 97.2  91.9  0.43| 23  22   1   0| 340  111  26  13|  87.9  76.2  88.6 
 76.2  88.2  1.94| 74  34  35   5| 848 1987 113  94|  64.7  58.5  66.0 
 58.3  92.6  0.64| 44  13  16  15| 153 1358  33  31|  52.6  62.9  53.6 
 76.4  92.6  0.66| 36  24   8   4| 159  609  11  10|  69.9  70.3  70.2 
 48.0  87.2  1.31| 43  12  15  16| 264 1949  26  28|  40.3  58.7  41.0 
 73.4  79.4  0.75|  9   4   5   0| 135  188   1   4|  54.2  58.6  54.3 
 ----------------------------------------------------------------------
 71.6  88.6  0.96| 38  18  13   7| 317 1034  35  30|  61.6  64.2  62.3 

* * * * * * * * * * *
Comparison to H2T...
S2L1, S2L2, S2L3, S1L1-1, S1L1-2

--- H2T results...
 Rcll  Prcn   FAR| GT  MT  PT  ML|  FP   FN IDs  FM|  MOTA  MOTP MOTAL 
 94.4  98.4  0.08| 23  22   1   0|  62  222   5  10|  92.7  72.9  92.8 
 71.0  90.7  1.40| 74  29  41   4| 609 2425 118 172|  62.3  52.7  63.7 
 60.8  93.0  0.62| 44  13  20  11| 150 1279  35  37|  55.1  53.5  56.1 
 41.4  99.7  0.02| 46  11  16  19|   5 2238  11  10|  41.0  75.7  41.3 
 58.5  97.8  0.14| 36  18  10   8|  34 1073   4   8|  57.0  58.9  57.1 
 ----------------------------------------------------------------------
 65.2  95.9  0.45| 45  19  18   8| 172 1447  35  47|  61.6  62.7  62.2 

--- Our results...
 Rcll  Prcn   FAR| GT  MT  PT  ML|  FP   FN IDs  FM|  MOTA  MOTP MOTAL 
 97.2  91.9  0.43| 23  22   1   0| 340  111  26  13|  87.9  76.2  88.6 
 76.2  88.2  1.94| 74  34  35   5| 848 1987 113  94|  64.7  58.5  66.0 
 58.3  92.6  0.64| 44  13  16  15| 153 1358  33  31|  52.6  62.9  53.6 
 58.2  90.6  1.04| 46  18  19   9| 230 1598  28  26|  51.4  64.4  52.1 
 76.4  92.6  0.66| 36  24   8   4| 159  609  11  10|  69.9  70.3  70.2 
 ----------------------------------------------------------------------
 73.3  91.2  0.94| 45  22  16   7| 346 1133  42  35|  65.3  66.5  66.1


Published with MATLAB® R2015a