main.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 简介\n",
    "本项目是参加飞桨常规赛：中文场景文字识别的项目，项目score为85.68143。\n",
    "\n",
    "生成的预测文件为/home/aistudio/result.txt\n",
    "\n",
    "项目任务为识别包含中文文字的街景图片，准确识别图片中的文字\n",
    "\n",
    "本项目源于\n",
    "https://aistudio.baidu.com/aistudio/projectdetail/681670\n",
    "\n",
    "感谢开发者为开源社区做出的贡献"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 赛题说明\n",
    "**赛题背景**\n",
    "\n",
    "中文场景文字识别技术在人们的日常生活中受到广泛关注，具有丰富的应用场景，如：拍照翻译、图像检索、场景理解等。然而，中文场景中的文字面临着包括光照变化、低分辨率、字体以及排布多样性、中文字符种类多等复杂情况。如何解决上述问题成为一项极具挑战性的任务。\n",
    "\n",
    "本次飞桨常规赛以 中文场景文字识别 为主题，由2019第二届中国AI+创新创业全国大赛降低难度而来，提供大规模的中文场景文字识别数据，旨在为研究者提供学术交流平台，进一步推动中文场景文字识别算法与技术的突破。\n",
    "\n",
    "**比赛任务**\n",
    "\n",
    "要求选手必须使用飞桨对图像区域中的文字行进行预测，返回文字行的内容。\n",
    "\n",
    "**数据集介绍**\n",
    "\n",
    "本次竞赛数据集共包括33万张图片，其中21万张图片作为训练集，12万张作为测试集。数据集采自中国街景，并由街景图片中的文字行区域（例如店铺标牌、地标等等）截取出来而形成。所有图像都经过一些预处理，将文字区域利用仿射变化，等比映射为一张高为48像素的图片，如下图1所示：\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/fb3cf59747e04f0cb9adde6a5a1945b3d9ef82f3b7c14c98bf248eb1c3886a3f)\n",
    "\n",
    "\n",
    "(a) 标注：魅派集成吊顶\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/57d58a35e1f34278bdb013b3f945ab69cddacf37c7fe40deba3c124fa1249753)\n",
    "\n",
    "\n",
    "(b) 标注：母婴用品连锁\n",
    "图1\n",
    "\n",
    "**标注文件**\n",
    "\n",
    "平台提供的标注文件为.txt文件格式。样例如下：\n",
    "\n",
    "\n",
    "\n",
    "| h | w | name | value |\n",
    "| -------- | -------- | -------- |-------- |\n",
    "| 128 | 48 | img_1.jpg | 文本1|\n",
    "| 56\t| 48\t| img_2.jpg|\t文本2|\n",
    "其中，文件中的四列分别是图片的宽、高、文件名和文字标注。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 安装第三方库\n",
    "注意：若项目重启后无法运行程序，尝试重新安装如下相应的依赖。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "! mkdir /home/aistudio/external-libraries\r\n",
    "! pip install --upgrade pip -t /home/aistudio/external-libraries\r\n",
    "! pip install tqdm paddlepaddle-gpu==1.7.1.post97 -i https://mirror.baidu.com/pypi/simple -t /home/aistudio/external-libraries\r\n",
    "! pip install pqi -t /home/aistudio/external-libraries\r\n",
    "! pqi use aliyun -t /home/aistudio/external-libraries\r\n",
    "! pip install tqdm imgaug lmdb matplotlib opencv-python Pillow python-Levenshtein PyYAML trdg anyconfig -t /home/aistudio/external-libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 处理数据集\n",
    "* 项目重启后，必须要重新解压数据，压缩包内含训练集图片、训练集图片信息、测试集图片， 否则无法运行本程序\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('/home/aistudio/external-libraries')\n",
    "import os\n",
    "os.chdir('/home/aistudio/data/data10879')\n",
    "! rm -rf /test_images/\n",
    "os.chdir('/home/aistudio/work')\n",
    "! unzip test_images.zip\n",
    "! mv /home/aistudio/work/test_images/ /home/aistudio/data/data10879\n",
    "! rm -rf /home/aistudio/work/__MACOSX/\n",
    "! rm -rf /home/aistudio/work/test_images/\n",
    "os.chdir('/home/aistudio/data/data10879')\n",
    "! rm -rf /train_img/\n",
    "! tar -zxf train_img.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 预处理\n",
    "\n",
    "* 项目重启后，需要重新做数据预处理\n",
    "* 文件 langconv(language convert)，这个文件用来把繁体字转成简体字<br>\n",
    "\n",
    "* 函数 read_ims_list：读取train.list文件，生成图片的信息字典\n",
    "* 函数 modify_ch：对标签label进行修改，进行四项操作，分别是“繁体->简体”、“大写->小写”、“删除空格”、“删除符号”。\n",
    "* 函数 pipeline：调用定义的函数，对训练数据进行初步处理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from work.langconv import Converter\n",
    "import codecs\n",
    "import random\n",
    "import sys\n",
    "import os\n",
    "from os.path import join as pjoin\n",
    "\n",
    "os.chdir('/home/aistudio')\n",
    "sys.path.append('/home/aistudio/work')\n",
    "def read_ims_list(path_ims_list):\n",
    "    \"\"\"\n",
    "    读取 train.list 文件\n",
    "    \"\"\"\n",
    "    ims_info_dic = {}\n",
    "    with open(path_ims_list, 'r', encoding='utf-8') as f:\n",
    "        for line in f:\n",
    "            parts = line.strip().split(maxsplit=3)\n",
    "            w, h, file, label = parts[0], parts[1], parts[2], parts[3]\n",
    "            ims_info_dic[file] = {'label': label, 'w': int(w)}\n",
    "    return ims_info_dic\n",
    "    \n",
    "\n",
    "def modify_ch(label):\n",
    "    # 繁体 -> 简体\n",
    "    label = Converter(\"zh-hans\").convert(label)\n",
    "\n",
    "    # 大写 -> 小写\n",
    "    label = label.lower()\n",
    "\n",
    "    # 删除空格\n",
    "    label = label.replace(' ', '')\n",
    "\n",
    "    # 删除符号\n",
    "    for ch in label:\n",
    "        if (not '\\u4e00' <= ch <= '\\u9fff') and (not ch.isalnum()):\n",
    "            label = label.replace(ch, '')\n",
    "\n",
    "    return label\n",
    "\n",
    "def save_txt(data, file_path):\n",
    "    \"\"\"\n",
    "    将一个list的数组写入txt文件里\n",
    "    :param data:\n",
    "    :param file_path:\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    if not isinstance(data, list):\n",
    "        data = [data]\n",
    "    with open(file_path, mode='w', encoding='utf8') as f:\n",
    "        f.write('\\n'.join(data))\n",
    "\n",
    "def pipeline(dataset_dir):\n",
    "    path_ims        = pjoin(dataset_dir, \"train_images\")\n",
    "    path_ims_list   = pjoin(dataset_dir, \"train.list\")\n",
    "    path_train_list = pjoin('/home/aistudio/work', \"train.txt\")\n",
    "    path_test_list  = pjoin('/home/aistudio/work', \"test.txt\")\n",
    "    path_label_list = pjoin('/home/aistudio/work', \"dict.txt\")\n",
    "\n",
    "    # 读取数据信息\n",
    "    file_info_dic = read_ims_list(path_ims_list)\n",
    "\n",
    "    # 创建 train.txt\n",
    "    class_set = set()\n",
    "    data_list = []\n",
    "    for file, info in file_info_dic.items():\n",
    "        label = info['label']\n",
    "        label = modify_ch(label)\n",
    "\n",
    "        # 异常: 标签为空\n",
    "        if label == '':\n",
    "            continue\n",
    "\n",
    "        for e in label:\n",
    "            class_set.add(e)\n",
    "        data_list.append(\"{0}\\t{1}\".format(pjoin('/home/aistudio/',path_ims, file), label))\n",
    "        \n",
    "    # 创建 label_list.txt\n",
    "    class_list = list(class_set)\n",
    "    class_list.sort()\n",
    "    print(\"class num: {0}\".format(len(class_list)))\n",
    "    with codecs.open(path_label_list, \"w\", encoding='utf-8') as label_list:\n",
    "        for id, c in enumerate(class_list):\n",
    "            # label_list.write(\"{0}\\t{1}\\n\".format(c, id))\n",
    "            label_list.write(\"{0}\\n\".format(c))\n",
    "\n",
    "    # 随机切分\n",
    "    random.shuffle(data_list)\n",
    "    val_len = int(len(data_list) * 0.05)\n",
    "    val_list = data_list[-val_len:]\n",
    "    train_list = data_list[:-val_len]\n",
    "    print('训练集数量: {}, 验证集数量: {}'.format(len(train_list),len(val_list)))\n",
    "    save_txt(train_list,path_train_list)\n",
    "    save_txt(val_list,path_test_list)\n",
    "    \n",
    "random.seed(0)\n",
    "pipeline(dataset_dir=\"data/data10879\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 对于PaddleOCR提供的配置文件rec_r34_vd_none_bilstm_ctc.yml我做出了如下修改\n",
    "\n",
    "1.将epoch_num改为30\n",
    "\n",
    "2.将train_batch_size_per_card改为256\n",
    "\n",
    "3.将test_batch_size_per_card改为128\n",
    "\n",
    "4.将base_lr改为0.00001\n",
    "\n",
    "经测试这样能提高score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 启动训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\r\n",
    "os.chdir('/home/aistudio/work/PaddleOCR/')\r\n",
    "! pwd\r\n",
    "! export PYTHONPATH=$PYTHONPATH:.\r\n",
    "! python tools/train.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 启动测试\n",
    "***注意,没有模型文件的情况下请先进行训练再进行测试***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\r\n",
    "os.chdir('/home/aistudio/work/PaddleOCR/')\r\n",
    "! pwd\r\n",
    "! export PYTHONPATH=$PYTHONPATH:.\r\n",
    "! python tools/infer_rec.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml -o Global.checkpoints=output/rec_CRNN_aug_341/latest\r\n",
    "#! python tools/export_model.py -c \"configs/rec/rec_r34_vd_none_bilstm_ctc.yml\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# **方案说明书**\n",
    "# 1. 比赛介绍+赛题重点难点剖析\n",
    "\n",
    "* 比赛介绍：本项目是参加飞桨常规赛：中文场景文字识别的项目，项目任务为识别包含中文文字的街景图片，准确识别图片中的文字。\n",
    "\n",
    "* 赛题重点难点剖析：中文场景中的文字面临着包括光照变化、低分辨率、字体以及排布多样性、中文字符种类多等复杂情况。如何解决上述问题成为一项极具挑战性的任务。\n",
    "\n",
    "# 2.  思路介绍+方案亮点\n",
    "* 思路介绍：第一步是针对中文场景下的数据预处理（包括：把繁体字转成简体字，大写->小写，删除空格，删除符号等操作），结合相应的中文字典来提升文字识别的准确率。第二步是在飞桨框架下采用当前业界最经典的CRNN算法架构来建模与求解，以保证模型的性能。\n",
    "\n",
    "* 方案亮点：结合中文场景下的字典资源来完成数据的预处理，可以更好的构建训练模型的语料;使用了飞桨的PaddleOCR模块,通过Fine-Tune预训练模型的方式达到快速训练和预测并且获得很好的预测效果\n",
    "\n",
    "# 3. 具体方案分享\n",
    "使用了飞桨PaddleOCR提供的rec_r34_vd_none_bilstm_ctc.yml配置文件,修改如下:\n",
    "```\n",
    "Global:\n",
    "  algorithm: CRNN\n",
    "  use_gpu: true\n",
    "  epoch_num: 30\n",
    "  log_smooth_window: 20\n",
    "  print_batch_step: 100\n",
    "  save_model_dir: output/rec_CRNN_aug_341\n",
    "  save_epoch_step: 1\n",
    "  eval_batch_step: 1800\n",
    "  train_batch_size_per_card: 256\n",
    "  test_batch_size_per_card: 128\n",
    "  image_shape: [3, 32, 256]\n",
    "  max_text_length: 64\n",
    "  character_type: ch\n",
    "  loss_type: ctc\n",
    "  reader_yml: ./configs/rec/rec_icdar15_reader.yml\n",
    "  pretrain_weights: /home/aistudio/work/PaddleOCR/model/latest\n",
    "  checkpoints: /home/aistudio/work/PaddleOCR/output/rec_CRNN_aug_341/latest\n",
    "  save_inference_dir: /home/aistudio/work/test\n",
    "  character_dict_path: /home/aistudio/work/dict.txt\n",
    "  infer_img: /home/aistudio/data/data10879/test_images\n",
    "  \n",
    "Architecture:\n",
    "  function: ppocr.modeling.architectures.rec_model,RecModel\n",
    "\n",
    "Backbone:\n",
    "  function: ppocr.modeling.backbones.rec_resnet_vd,ResNet\n",
    "  layers: 34\n",
    " \n",
    "Head:\n",
    "  function: ppocr.modeling.heads.rec_ctc_head,CTCPredict\n",
    "  encoder_type: rnn\n",
    "  SeqRNN:\n",
    "    hidden_size: 256\n",
    "    \n",
    "Loss:\n",
    "  function: ppocr.modeling.losses.rec_ctc_loss,CTCLoss\n",
    "\n",
    "Optimizer:\n",
    "  function: ppocr.optimizer,AdamDecay\n",
    "  base_lr: 0.00001\n",
    "  beta1: 0.9\n",
    "  beta2: 0.999\n",
    "```\n",
    "\n",
    "# 4. 模型应用结果分析\n",
    "* 不同模型结果的对比分析：\n",
    "\n",
    "| 模型名称 | score | norm_distance |word_acc |\n",
    "| -------- | -------- | -------- | -------- |\n",
    "| 官方基线模型（PaddleOCR：中文场景文字识别）| 82.87     | 0.93946     | 0.82872     |\n",
    "|CRNN rec_r34_vd_none_bilstm_ctc | 85.68143    | 0.95619    | 0.85681     |\n",
    "* 调参优化过程分析\n",
    "1. 将epoch_num改为30 \n",
    "2. 将train_batch_size_per_card改为256 \n",
    "3. 将test_batch_size_per_card改为128 \n",
    "4. 将base_lr改为0.00001 经测试这样能提高score\n",
    "\n",
    "# 5.  总结+改进完善方向\n",
    "* 总结 \n",
    " 第一次参加OCR方面的比赛,扩宽了自己的眼界,使自己对计算机视觉、深度学习模型等有了更深刻的认识,学习了很多调整参数的技巧,阅读了很多国内外的相关论文文献,本次比赛的模型可以用在中文场景文字识别的实际应用中.\n",
    "* 改进完善方向\n",
    "1.  CNN部分目前用的RESNET，后续可以考虑改成VGG网络；\n",
    "2.  可以采取本地生成数据集的方式进行数据增强\n",
    "3.  后续可以进一步优化损失函数和训练策略，以便提升模型的收敛速度。\n",
    "\n",
    "#  7.  参考资料\n",
    "1. EAST:\n",
    "@inproceedings{zhou2017east,\n",
    "  title={EAST: an efficient and accurate scene text detector},\n",
    "  author={Zhou, Xinyu and Yao, Cong and Wen, He and Wang, Yuzhi and Zhou, Shuchang and He, Weiran and Liang, Jiajun},\n",
    "  booktitle={Proceedings of the IEEE conference on Computer Vision and Pattern Recognition},\n",
    "  pages={5551--5560},\n",
    "  year={2017}\n",
    "}\n",
    "\n",
    "2. DB:\n",
    "@article{liao2019real,\n",
    "  title={Real-time Scene Text Detection with Differentiable Binarization},\n",
    "  author={Liao, Minghui and Wan, Zhaoyi and Yao, Cong and Chen, Kai and Bai, Xiang},\n",
    "  journal={arXiv preprint arXiv:1911.08947},\n",
    "  year={2019}\n",
    "}\n",
    "\n",
    "3. DTRB:\n",
    "@inproceedings{baek2019wrong,\n",
    "  title={What is wrong with scene text recognition model comparisons? dataset and model analysis},\n",
    "  author={Baek, Jeonghun and Kim, Geewook and Lee, Junyeop and Park, Sungrae and Han, Dongyoon and Yun, Sangdoo and Oh, Seong Joon and Lee, Hwalsuk},\n",
    "  booktitle={Proceedings of the IEEE International Conference on Computer Vision},\n",
    "  pages={4715--4723},\n",
    "  year={2019}\n",
    "}\n",
    "\n",
    "4. SAST:\n",
    "@inproceedings{wang2019single,\n",
    "  title={A Single-Shot Arbitrarily-Shaped Text Detector based on Context Attended Multi-Task Learning},\n",
    "  author={Wang, Pengfei and Zhang, Chengquan and Qi, Fei and Huang, Zuming and En, Mengyi and Han, Junyu and Liu, Jingtuo and Ding, Errui and Shi, Guangming},\n",
    "  booktitle={Proceedings of the 27th ACM International Conference on Multimedia},\n",
    "  pages={1277--1285},\n",
    "  year={2019}\n",
    "}\n",
    "\n",
    "5. SRN:\n",
    "@article{yu2020towards,\n",
    "  title={Towards Accurate Scene Text Recognition with Semantic Reasoning Networks},\n",
    "  author={Yu, Deli and Li, Xuan and Zhang, Chengquan and Han, Junyu and Liu, Jingtuo and Ding, Errui},\n",
    "  journal={arXiv preprint arXiv:2003.12294},\n",
    "  year={2020}\n",
    "}\n",
    "\n",
    "6. end2end-psl:\n",
    "@inproceedings{sun2019chinese,\n",
    "  title={Chinese Street View Text: Large-scale Chinese Text Reading with Partially Supervised Learning},\n",
    "  author={Sun, Yipeng and Liu, Jiaming and Liu, Wei and Han, Junyu and Ding, Errui and Liu, Jingtuo},\n",
    "  booktitle={Proceedings of the IEEE International Conference on Computer Vision},\n",
    "  pages={9086--9095},\n",
    "  year={2019}\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}