RefTeacher

RefTeacher is a strong baseline method for Semi-Supervised Referring Expression Comprehension. This is the official implementations of our paper "RefTeacher: A Strong Baseline for Semi-Supervised Referring Expression Comprehension", including models such as RealGIN and SimREC.

Updates

• (2023/5/23) Release our RefTeacher project.

Installation

• Clone this repo

git clone https://github.com/Disguiser15/RefTeacher.git
cd RefTeacher

• Create a conda virtual environment and activate it

conda create -n refteacher python=3.7 -y
conda activate refteacher

• Install Pytorch following the official installation instructions
• Install mmcv following the installation guide
• Install Spacy and initialize the GloVe and install other requirements as follows:

pip install -r requirements.txt
wget https://github.com/explosion/spacy-models/releases/download/en_vectors_web_lg-2.1.0/en_vectors_web_lg-2.1.0.tar.gz -O en_vectors_web_lg-2.1.0.tar.gz
pip install en_vectors_web_lg-2.1.0.tar.gz

Data preparation

•  Follow the instructions of DATA_PRE_README.md to generate training data and testing data.
•  Download the pretrained weights of backbone (vgg, darknet, cspdarknet, DResNet, etc.). Expect for DResNet, all pretrained backbones are trained on COCO 2014 train+val set while removing the images appeared in the val+test sets of RefCOCO, RefCOCO+ and RefCOCOg (nearly 6500 images). Please follow the instructions of DATA_PRE_README.md to download them.

Training and Evaluation

1. Config preparation. Prepare your own configs in configs. We provided config files including Supervised baseline, STAC baseline and RefTeacher.

2. Train the model. Run bash/train.sh under the corresponding directory in scripts.

• Train Supervised baseline.

bash scripts/sup/train.sh config/sup/realgin_sup_baseline.yaml None

• Train STAC baseline.

bash scripts/stac/train.sh config/stac/realgin_semi_stac.yaml /path/to/pretrained/checkpoint.pth

Setting /path/to/pretrained/checkpoint.pth to the specific checkpoint.pth you want to load as the teacher model.

• Train our RefTeacher.

# RealGIN
bash scripts/refteacher/train.sh config/refteacher/realgin_semi_refteacher.yaml
# SimREC
bash scripts/refteacher/train.sh config/refteacher/simrec_semi_refteacher.yaml

The training logs, tensorboard logs and model checkpoints will be automatically saved under LOG_PATH set in config.

3. Test the model. Run bash/eval.sh in scripts to evaluate the saved teacher or student weight.

# RealGIN
bash scripts/eval/eval.sh config/refteacher/realgin_semi_refteacher.yaml /path/to/checkpoint
# SimREC
bash scripts/eval/eval.sh config/refteacher/simrec_semi_refteacher.yaml /path/to/checkpoint

License

This project is released under the Apache 2.0 license.

Citation

If this repository is helpful for your research, or you want to refer the provided results in your paper, consider cite:

@InProceedings{Sun_2023_CVPR,
    author    = {Sun, Jiamu and Luo, Gen and Zhou, Yiyi and Sun, Xiaoshuai and Jiang, Guannan and Wang, Zhiyu and Ji, Rongrong},
    title     = {RefTeacher: A Strong Baseline for Semi-Supervised Referring Expression Comprehension},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {19144-19154}
}

@article{luo2022what,
  title={What Goes beyond Multi-modal Fusion in One-stage Referring Expression Comprehension: An Empirical Study},
  author={Luo, Gen and Zhou, Yiyi and Sun, Jiamu and Huang, Shubin and Sun, Xiaoshuai and Ye, Qixiang and Wu, Yongjian and Ji, Rongrong},
  journal={arXiv preprint arXiv:2204.07913},
  year={2022}
}

Acknowledgement

Thanks a lot for the nicely organized code from the following repos

• OpenVQA.
• Swin-Transformer
• Detectron2