Initial commit

.gitignore

+__pycache__
+.idea
+venv
\ No newline at end of file

README.md

-# DriPE dataset
+<h1 style="text-align:center">
+DriPE: A Dataset for Human Pose Estimation in Real-World Driving Settings
+</h1>
+<div style="text-align:center">
+<h3>
+<a href="https://liris.cnrs.fr/page-membre/romain-guesdon">Romain Guesdon</a>,
+<a href="https://liris.cnrs.fr/page-membre/carlos-crispim-junior">Carlos Crispim-Junior</a>,
+<a href="https://liris.cnrs.fr/page-membre/laure-tougne">Laure Tougne</a>
+<br>
+<br>
+ICCV: International Conference on Computer Vision 2021
+<br>
+Workshop AVVision : Autonomous Vehicle Vision
+</h3>
+</div>
 
-This webpage will be updated soon with more information on how to download the DriPE dataset.
+# Table of content
+- [Overview](#overview)
+- [Dataset](#dataset)
+- [Evaluation](#evaluation)
+- [Training](#training)
+- [Citation](#citation)
+- [Acknowledgements](#acknowledgements)
 
+# Overview
+This repository contains link to download the DriPE dataset,
+along with trained weights for the three networks presented in this paper: 
+SBl, MSPN and RSN.
+Furthermore, we provide the code to evaluate HPE networks with mAPK metric, our keypoint-centered metric.
+
+# Dataset
+DriPE dataset can be found [here](). We provide the 10k images, 
+along with keypoint annotations, split as:
+* 6.4k for training
+* 1.3k for validation
+* 1.3k for testing
+
+Annotations follow the COCO annotation style, with 17 keypoints. 
+More information can be found [here](https://cocodataset.org/#format-data).
+
+
+# Networks
+We used in our study three architectures:
+* __SBl__: Simple Baselines for Human Pose Estimation and Tracking (Xiao 2018) [GitHub](https://github.com/microsoft/human-pose-estimation.pytorch)
+* __MSPN__: Rethinking on Multi-Stage Networks for Human Pose Estimation (Li 2019) [GitHub](https://github.com/megvii-detection/MSPN)
+* __RSN__: Learning Delicate Local Representations for Multi-Person Pose Estimation (Cai 2020) [GitHub](https://github.com/caiyuanhao1998/RSN)
+
+We used for training and for inference the code provided by the authors in the three linked repositories.
+Weights of the trained model evaluated in our study can be found [here]().
+More details about the training can be found in our paper.
+
+# Evaluation
+Evaluation is performed using two metrics:
+* __AP OKS__, the original metric from COCO dataset, which is already implemented in the [cocoapi](https://github.com/cocodataset/cocoapi)
+and in the three network repositories
+* __mAPK__, our new keypoint-centered metric. We provide script for evaluate the network predictions in this repository.
+
+Evaluation with mAPK can be used by running the eval_mpk.py script.
+```Script to evaluate prediction in COCO format using the mAPK metric.
+Usage: python eval_mapk.py [json_prediction_path] [json_annotation_path]
+Paths can be absolute, relative to the script or relative to the respective json/gts or json/preds directory.
+    -h, --help\tdisplay this help message and exit
+```
+
+We provide in this repo one annotation file and one prediction. To evaluate these predictions, run:
+```
+python eval_mapk.py keypoints_autob_coco_test_results.json autob_coco_test.json
+```
+
+Expected results are :
+	F1 score: 0.9278493428365919
+    |  Head  |  Should.  |  Elbow  |  Wrist  |  Hip  |  Knee  |  Ankle  |  All  |  Mean  |  Std
+:-- | :------: | :---------: | :-------: | :-------: | :-----: | :------: | :-------: | :-----: | :------: | :-----:
+AP  |  0.84  |     0.90  |   0.94  |   0.96   0.98  |  0.95  |   0.68  |  0.91  |  0.90  |  0.10
+AR  |  0.87  |     0.96  |   0.96  |   0.97   0.98  |  0.95  |   0.80  |  0.94  |  0.93  |  0.06
+
+# Citation
+If you use this dataset or code in your research, please cite the paper:
+```
+@InProceedings{Guesdon_2021_ICCV,
+    author    = {Guesdon, Romain and Crispim-Junior, Carlos and Tougne, Laure},
+    title     = {DriPE: A Dataset for Human Pose Estimation in Real-World Driving Settings},
+    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops},
+    month     = {October},
+    year      = {2021},
+    pages     = {2865-2874}
+}
+```
+
+# Acknowledgments
\ No newline at end of file

eval_mapk.py

+import os.path
+import sys
+from pycocotools.coco import COCO
+from pycocotools.cocoeval import COCOeval
+import numpy as np
+from statistics import stdev
+from collections import OrderedDict
+from tabulate import tabulate
+
+COCO_KP = {
+    'keypoints': ['nose', 'left_eye', 'right_eye', 'left_ear', 'right_ear', 'left_shoulder', 'right_shoulder',
+                  'left_elbow', 'right_elbow', 'left_wrist', 'right_wrist', 'left_hip', 'right_hip', 'left_knee',
+                  'right_knee', 'left_ankle', 'right_ankle'],
+    'skeleton': [[16, 14], [14, 12], [17, 15], [15, 13], [12, 13], [6, 12], [7, 13], [6, 7], [6, 8], [7, 9], [8, 10],
+                 [9, 11], [2, 3], [1, 2], [1, 3], [2, 4], [3, 5], [4, 6], [5, 7]],
+    'parts': OrderedDict(
+        [('Head', [0, 1, 2, 3, 4]), ('Should.', [5, 6]), ('Elbow', [7, 8]), ('Wrist', [9, 10]), ('Hip', [11, 12]),
+         ('Knee', [13, 14]), ('Ankle', [15, 16]), ('All', list(range(17)))]),
+}
+
+
+def get_coco_eval(gts_name, dts_name, one_range=True):
+    coco_gt = COCO(gts_name)
+    coco_dt = coco_gt.loadRes(dts_name)
+
+    imgIds = sorted(coco_gt.getImgIds())
+    # running evaluation
+    cocoEval = COCOeval(coco_gt, coco_dt, 'keypoints')
+    cocoEval.params.imgIds = imgIds
+    if one_range:
+        cocoEval.params.areaRng = [cocoEval.params.areaRng[0]]
+        cocoEval.params.areaRngLbl = [cocoEval.params.areaRngLbl[0]]
+    cocoEval.evaluate()
+
+    return cocoEval
+
+
+def comp_mets(coco, site_thold=0.25, dist_thold=1 / 8, fix_scale=False, weights=None):
+    N_KP = 17
+    coco.params.iouThrs = np.array([0.])
+    coco.evaluate()
+    eval_imgs = coco.evalImgs
+
+    tp = [0 for _ in range(N_KP)]
+    fp = [0 for _ in range(N_KP)]
+    fn = [0 for _ in range(N_KP)]
+    d2s = [[] for _ in range(N_KP)]
+
+    ref_p = [0 for _ in range(N_KP)]
+    tp_conf = []
+    fp_conf = []
+
+    d2_arms = []
+    arm = (12, 6)
+
+    sigmas = np.array(
+        [.26, .25, .25, .35, .35, .79, .79, .72, .72, .62, .62, 1.07, 1.07, .87, .87, .89, .89]) / 10.0
+    vars = (sigmas * 2) ** 2
+
+    for img in eval_imgs:
+        if img is None:
+            continue
+        if fix_scale:
+            img_an = coco.cocoGt.loadImgs(int(img['image_id']))[0]
+            scale = min(img_an['height'], img_an['width'])
+
+        for g, gt_id in enumerate(img['gtIds']):
+            if img['gtIgnore'][g]:
+                continue
+            dt_id = int(img['gtMatches'][0, g])
+
+            # Handle unmatched GT
+            if not dt_id:
+                gt_an = coco.cocoGt.loadAnns(gt_id)[0]
+                for kp in range(N_KP):
+                    if gt_an['keypoints'][3 * kp + 2] > 0:
+                        fn[kp] += 1
+                continue
+
+            gt_an = coco.cocoGt.loadAnns(gt_id)[0]
+            dt_an = coco.cocoDt.loadAnns(dt_id)[0]
+
+            if not fix_scale:
+                scale = gt_an['area'] ** 0.5
+
+            if gt_an['keypoints'][3 * arm[0] + 2] * gt_an['keypoints'][3 * arm[1] + 2]:
+                d2el = np.array(gt_an['keypoints'][3 * arm[0]:3 * arm[0] + 2]) - np.array(
+                    gt_an['keypoints'][3 * arm[1]:3 * arm[1] + 2])
+                d2el = np.sum(d2el ** 2)
+                d2_arms.append(np.sqrt(d2el) / scale)
+
+            for kp in range(N_KP):
+                has_gt = gt_an['keypoints'][3 * kp + 2] > 0
+                has_dt = dt_an['keypoints'][3 * kp + 2] >= site_thold
+
+                if not (has_gt and has_dt):
+                    fn[kp] += int(has_gt)
+                    fp[kp] += int(has_dt)
+                    continue
+
+                gt = np.array(gt_an['keypoints'][3 * kp:3 * kp + 2])
+                dt = np.array(dt_an['keypoints'][3 * kp:3 * kp + 2])
+
+                # Distance between gt and dt. Shape |gt|,|dt|
+                d2 = np.sum(np.array(dt - gt) ** 2)
+                dist = d2 / (scale ** 2 * vars[kp] + np.spacing(1))
+
+                if np.exp(-dist / 2) > dist_thold:
+                    tp[kp] += 1
+                    tp_conf.append(dt_an['keypoints'][3 * kp + 2])
+                else:
+                    fn[kp] += 1
+                    fp[kp] += 1
+                    fp_conf.append(dt_an['keypoints'][3 * kp + 2])
+                    ref_p[kp] += 1
+
+                d2s[kp].append(dist * vars[kp])
+
+        # Handle unmatched DT
+
+        if len(img['dtIds']) > len(img['dtMatches'][0]):
+            for dt_id in img['dtIds']:
+                dt_an = coco.cocoGt.loadAnns(dt_id)[0]
+                for kp in range(N_KP):
+                    if dt_an['keypoints'][3 * kp + 2] > site_thold:
+                        fp[kp] += 1
+
+    means = [sum(d2l) / len(d2l) if len(d2l) else -1 for d2l in d2s]
+    stds = [stdev(d2l) if len(d2l) >= 2 else -1 for d2l in d2s]
+
+    ref_p = np.array(ref_p)
+
+    return tp, fp, fn, means, stds, ref_p
+
+
+def compute_apr(tp, fp, fn, means, stds):
+    tp = np.array(tp)
+    fp = np.array(fp)
+    fn = np.array(fn)
+    stds = np.array(stds)
+
+    # Remove -1 from stds:
+    stds[stds <= 0] = stds[stds > 0].mean()
+
+    aps = []
+    ars = []
+
+    for parts in COCO_KP['parts'].values():
+        if (tp[parts] + fp[parts]).any():
+            ap = np.sum(tp[parts]) / np.sum(tp[parts] + fp[parts])
+        else:
+            ap = -1
+
+        if (tp[parts] + fn[parts]).any():
+            ar = np.sum(tp[parts]) / np.sum(tp[parts] + fn[parts])
+        else:
+            ar = -1
+
+        aps.append(ap)
+        ars.append(ar)
+
+    m_w = np.ones((len(COCO_KP['parts']) - 1,))
+
+    aps_m = np.array(aps)
+    ars_m = np.array(ars)
+    aps.append(np.sum(aps_m[:-1] * m_w) / m_w.sum())
+    aps.append(np.std(aps_m[:-1] * m_w))
+    ars.append(np.sum(ars_m[:-1] * m_w) / m_w.sum())
+    ars.append(np.std(ars_m[:-1] * m_w))
+
+    return aps, ars
+
+
+def mean_ap(coco, site_thold=0.25, fix_scale=False):
+    aps, ars = [], []
+    tp, fp, fn = [], [], []
+    ref_p = []
+    for t in np.linspace(0.5, 0.95, 10):
+        res_coco = comp_mets(coco, site_thold=site_thold, dist_thold=t, fix_scale=fix_scale)
+        tp.append(res_coco[0])
+        fp.append(res_coco[1])
+        fn.append(res_coco[2])
+        ref_p.append(res_coco[5])
+
+        ap, ar = compute_apr(*res_coco[:-1])
+        aps.append(ap)
+        ars.append(ar)
+    aps = np.array(aps)
+    ars = np.array(ars)
+
+    tp = np.array(tp).sum(axis=0)
+    fp = np.array(fp).sum(axis=0)
+    fn = np.array(fn).sum(axis=0)
+
+    f1 = tp.sum() / (tp.sum() + 0.5 * (fp.sum() + fn.sum()))
+    print(f'\n\tF1 score: {f1:0.3f}')
+
+    map = aps.mean(axis=0)
+    mar = ars.mean(axis=0)
+
+    return map, mar
+
+
+def parse_args():
+    argv = sys.argv
+    argc = len(argv)
+
+    if argc < 2:
+        return
+
+    if argv[1] in ['--help', '-h']:
+        print('''Script to evaluate prediction in COCO format using the mAPK metric.
+Usage: python eval_mapk.py [json_prediction_path] [json_annotation_path]
+Paths can be absolute, relative to the script or relative to the respective json/gts or json/preds directory.
+    -h, --help\tdisplay this help message and exit
+        ''')
+
+        return
+
+    if argc >= 3:
+        gt_path = argv[2]
+        if not os.path.isfile(gt_path):
+            json_path = 'json/gts/' + gt_path
+            if os.path.isfile(json_path):
+                gt_path = json_path
+
+        pred_path = argv[1]
+        if not os.path.isfile(pred_path):
+            json_path = 'json/preds/' + pred_path
+            if os.path.isfile(json_path):
+                pred_path = json_path
+
+        coco_repo = get_coco_eval(gt_path, pred_path, one_range=True)
+        map, mar = mean_ap(coco_repo, site_thold=0.25, fix_scale=False)
+        print(
+            tabulate([['AP'] + map.tolist(), ['AR'] + mar.tolist()], headers=list(COCO_KP['parts']) + ['Mean', 'Std'],
+                     floatfmt=[None] + ['0.2f'] * len(map))
+        )
+
+
+if __name__ == '__main__':
+    parse_args()

requirements.txt

+numpy
+tabulate
\ No newline at end of file