## File: README.md Detectron2 is Facebook AI Research's next generation library that provides state-of-the-art detection and segmentation algorithms. It is the successor of [Detectron](https://github.com/facebookresearch/Detectron/) and [maskrcnn-benchmark](https://github.com/facebookresearch/maskrcnn-benchmark/). It supports a number of computer vision research projects and production applications in Facebook. ## Learn More about Detectron2 * Includes new capabilities such as panoptic segmentation, Densepose, Cascade R-CNN, rotated bounding boxes, PointRend, DeepLab, ViTDet, MViTv2 etc. * Used as a library to support building [research projects](projects/) on top of it. * Models can be exported to TorchScript format or Caffe2 format for deployment. * It [trains much faster](https://detectron2.readthedocs.io/notes/benchmarks.html). See our [blog post](https://ai.meta.com/blog/-detectron2-a-pytorch-based-modular-object-detection-library-/) to see more demos. See this [interview](https://ai.meta.com/blog/detectron-everingham-prize/) to learn more about the stories behind detectron2. ## Installation See [installation instructions](https://detectron2.readthedocs.io/tutorials/install.html). ## Getting Started See [Getting Started with Detectron2](https://detectron2.readthedocs.io/tutorials/getting_started.html), and the [Colab Notebook](https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5) to learn about basic usage. Learn more at our [documentation](https://detectron2.readthedocs.org). And see [projects/](projects/) for some projects that are built on top of detectron2. ## Model Zoo and Baselines We provide a large set of baseline results and trained models available for download in the [Detectron2 Model Zoo](MODEL_ZOO.md). ## License Detectron2 is released under the [Apache 2.0 license](LICENSE). ## Citing Detectron2 If you use Detectron2 in your research or wish to refer to the baseline results published in the [Model Zoo](MODEL_ZOO.md), please use the following BibTeX entry. ```BibTeX @misc{wu2019detectron2, author = {Yuxin Wu and Alexander Kirillov and Francisco Massa and Wan-Yen Lo and Ross Girshick}, title = {Detectron2}, howpublished = {\url{https://github.com/facebookresearch/detectron2}}, year = {2019} } ``` --- ## File: configs/common/README.md This directory provides definitions for a few common models, dataloaders, scheduler, and optimizers that are often used in training. The definition of these objects are provided in the form of lazy instantiation: their arguments can be edited by users before constructing the objects. They can be imported, or loaded by `model_zoo.get_config` API in users' own configs. --- ## File: configs/Detectron1-Comparisons/README.md Detectron2 model zoo's experimental settings and a few implementation details are different from Detectron. The differences in implementation details are shared in [Compatibility with Other Libraries](../../docs/notes/compatibility.md). The differences in model zoo's experimental settings include: * Use scale augmentation during training. This improves AP with lower training cost. * Use L1 loss instead of smooth L1 loss for simplicity. This sometimes improves box AP but may affect other AP. * Use `POOLER_SAMPLING_RATIO=0` instead of 2. This does not significantly affect AP. * Use `ROIAlignV2`. This does not significantly affect AP. In this directory, we provide a few configs that __do not__ have the above changes. They mimic Detectron's behavior as close as possible, and provide a fair comparison of accuracy and speed against Detectron. | [Faster R-CNN](faster_rcnn_R_50_FPN_noaug_1x.yaml) | 1x | 0.219 | 0.038 | 3.1 | 36.9 | | | 137781054 | [model](https://dl.fbaipublicfiles.com/detectron2/Detectron1-Comparisons/faster_rcnn_R_50_FPN_noaug_1x/137781054/model_final_7ab50c.pkl)   [metrics](https://dl.fbaipublicfiles.com/detectron2/Detectron1-Comparisons/faster_rcnn_R_50_FPN_noaug_1x/137781054/metrics.json) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [Keypoint R-CNN](keypoint_rcnn_R_50_FPN_1x.yaml) | 1x | 0.313 | 0.071 | 5.0 | 53.1 | | 64.2 | 137781195 | [model](https://dl.fbaipublicfiles.com/detectron2/Detectron1-Comparisons/keypoint_rcnn_R_50_FPN_1x/137781195/model_final_cce136.pkl)   [metrics](https://dl.fbaipublicfiles.com/detectron2/Detectron1-Comparisons/keypoint_rcnn_R_50_FPN_1x/137781195/metrics.json) | | [Mask R-CNN](mask_rcnn_R_50_FPN_noaug_1x.yaml) | 1x | 0.273 | 0.043 | 3.4 | 37.8 | 34.9 | | 137781281 | [model](https://dl.fbaipublicfiles.com/detectron2/Detectron1-Comparisons/mask_rcnn_R_50_FPN_noaug_1x/137781281/model_final_62ca52.pkl)   [metrics](https://dl.fbaipublicfiles.com/detectron2/Detectron1-Comparisons/mask_rcnn_R_50_FPN_noaug_1x/137781281/metrics.json) | ## Comparisons: * Faster R-CNN: Detectron's AP is 36.7, similar to ours. * Keypoint R-CNN: Detectron's AP is box 53.6, keypoint 64.2. Fixing a Detectron's [bug](https://github.com/facebookresearch/Detectron/issues/459) lead to a drop in box AP, and can be compensated back by some parameter tuning. * Mask R-CNN: Detectron's AP is box 37.7, mask 33.9. We're 1 AP better in mask AP, due to more correct implementation. See [this article](https://ppwwyyxx.com/blog/2021/Where-are-Pixels/) for details. For speed comparison, see [benchmarks](https://detectron2.readthedocs.io/notes/benchmarks.html). --- ## File: configs/quick_schedules/README.md These are quick configs for performance or accuracy regression tracking purposes. * `*instance_test.yaml`: can train on 2 GPUs. They are used to test whether the training can successfully finish. They are not expected to produce reasonable training results. * `*inference_acc_test.yaml`: They should be run using `--eval-only`. They run inference using pre-trained models and verify the results are as expected. * `*training_acc_test.yaml`: They should be trained on 8 GPUs. They finish in about an hour and verify the training accuracy is within the normal range. --- ## File: datasets/README.md # Use Builtin Datasets A dataset can be used by accessing [DatasetCatalog](https://detectron2.readthedocs.io/modules/data.html#detectron2.data.DatasetCatalog) for its data, or [MetadataCatalog](https://detectron2.readthedocs.io/modules/data.html#detectron2.data.MetadataCatalog) for its metadata (class names, etc). This document explains how to setup the builtin datasets so they can be used by the above APIs. [Use Custom Datasets](https://detectron2.readthedocs.io/tutorials/datasets.html) gives a deeper dive on how to use `DatasetCatalog` and `MetadataCatalog`, and how to add new datasets to them. Detectron2 has builtin support for a few datasets. The datasets are assumed to exist in a directory specified by the environment variable `DETECTRON2_DATASETS`. Under this directory, detectron2 will look for datasets in the structure described below, if needed. ``` $DETECTRON2_DATASETS/ coco/ lvis/ cityscapes/ VOC20{07,12}/ ``` You can set the location for builtin datasets by `export DETECTRON2_DATASETS=/path/to/datasets`. If left unset, the default is `./datasets` relative to your current working directory. The [model zoo](https://github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md) contains configs and models that use these builtin datasets. ## Expected dataset structure for [COCO instance/keypoint detection](https://cocodataset.org/#download): ``` coco/ annotations/ instances_{train,val}2017.json person_keypoints_{train,val}2017.json {train,val}2017/ # image files that are mentioned in the corresponding json ``` You can use the 2014 version of the dataset as well. Some of the builtin tests (`dev/run_*_tests.sh`) uses a tiny version of the COCO dataset, which you can download with `./datasets/prepare_for_tests.sh`. ## Expected dataset structure for PanopticFPN: Extract panoptic annotations from [COCO website](https://cocodataset.org/#download) into the following structure: ``` coco/ annotations/ panoptic_{train,val}2017.json panoptic_{train,val}2017/ # png annotations panoptic_stuff_{train,val}2017/ # generated by the script mentioned below ``` Install panopticapi by: ``` pip install git+https://github.com/cocodataset/panopticapi.git ``` Then, run `python datasets/prepare_panoptic_fpn.py`, to extract semantic annotations from panoptic annotations. ## Expected dataset structure for [LVIS instance segmentation](https://www.lvisdataset.org/dataset): ``` coco/ {train,val,test}2017/ lvis/ lvis_v0.5_{train,val}.json lvis_v0.5_image_info_test.json lvis_v1_{train,val}.json lvis_v1_image_info_test{,_challenge}.json ``` Install lvis-api by: ``` pip install git+https://github.com/lvis-dataset/lvis-api.git ``` To evaluate models trained on the COCO dataset using LVIS annotations, run `python datasets/prepare_cocofied_lvis.py` to prepare "cocofied" LVIS annotations. ## Expected dataset structure for [cityscapes](https://www.cityscapes-dataset.com/downloads/): ``` cityscapes/ gtFine/ train/ aachen/ color.png, instanceIds.png, labelIds.png, polygons.json, labelTrainIds.png ... val/ test/ # below are generated Cityscapes panoptic annotation cityscapes_panoptic_train.json cityscapes_panoptic_train/ cityscapes_panoptic_val.json cityscapes_panoptic_val/ cityscapes_panoptic_test.json cityscapes_panoptic_test/ leftImg8bit/ train/ val/ test/ ``` Install cityscapes scripts by: ``` pip install git+https://github.com/mcordts/cityscapesScripts.git ``` Note: to create labelTrainIds.png, first prepare the above structure, then run cityscapesescript with: ``` CITYSCAPES_DATASET=/path/to/abovementioned/cityscapes python cityscapesscripts/preparation/createTrainIdLabelImgs.py ``` These files are not needed for instance segmentation. Note: to generate Cityscapes panoptic dataset, run cityscapesescript with: ``` CITYSCAPES_DATASET=/path/to/abovementioned/cityscapes python cityscapesscripts/preparation/createPanopticImgs.py ``` These files are not needed for semantic and instance segmentation. ## Expected dataset structure for [Pascal VOC](http://host.robots.ox.ac.uk/pascal/VOC/index.html): ``` VOC20{07,12}/ Annotations/ ImageSets/ Main/ trainval.txt test.txt # train.txt or val.txt, if you use these splits JPEGImages/ ``` ## Expected dataset structure for [ADE20k Scene Parsing](http://sceneparsing.csail.mit.edu/): ``` ADEChallengeData2016/ annotations/ annotations_detectron2/ images/ objectInfo150.txt ``` The directory `annotations_detectron2` is generated by running `python datasets/prepare_ade20k_sem_seg.py`. --- ## File: demo/README.md ## Detectron2 Demo We provide a command line tool to run a simple demo of builtin configs. The usage is explained in [GETTING_STARTED.md](../GETTING_STARTED.md). See our [blog post](https://ai.facebook.com/blog/-detectron2-a-pytorch-based-modular-object-detection-library-) for a high-quality demo generated with this tool. --- ## File: detectron2/data/datasets/README.md ### Common Datasets The dataset implemented here do not need to load the data into the final format. It should provide the minimal data structure needed to use the dataset, so it can be very efficient. For example, for an image dataset, just provide the file names and labels, but don't read the images. Let the downstream decide how to read. --- ## File: detectron2/export/README.md This directory contains code to prepare a detectron2 model for deployment. Currently it supports exporting a detectron2 model to TorchScript, ONNX, or (deprecated) Caffe2 format. Please see [documentation](https://detectron2.readthedocs.io/tutorials/deployment.html) for its usage. ### Acknowledgements Thanks to Mobile Vision team at Facebook for developing the Caffe2 conversion tools. Thanks to Computing Platform Department - PAI team at Alibaba Group (@bddpqq, @chenbohua3) who help export Detectron2 models to TorchScript. Thanks to ONNX Converter team at Microsoft who help export Detectron2 models to ONNX. --- ## File: detectron2/layers/csrc/README.md To add a new Op: 1. Create a new directory 2. Implement new ops there 3. Delcare its Python interface in `vision.cpp`. --- ## File: detectron2/projects/README.md Projects live in the [`projects` directory](../../projects) under the root of this repository, but not here.