基于轻量化及多尺度特征提取的城市街景语义分割

A Semantic Segmentation Algorithm for Urban Streetscape Based on Lightweight and Multi-scale Feature Extraction

摘要

目前在移动端或者嵌入式设备等平台部署高性能神经网络算法是一大趋势.针对常规分割模型复杂度较高的问题,本文提出了一种基于轻量化及多尺度特征提取的城市街景语义分割算法.该算法采用轻量级网络MobileNetV3作为特征提取网络,引入特征金字塔网络在其横向连接处嵌入混洗注意力,加强信息融合,接着在其深层特征后堆叠三次深度不对称门控模块,进行局部信息和上下文信息的交互;然后采用多尺度膨胀金字塔模块,扩大感受野,获取多尺度特征信息,最后将不同层次的信息拼接融合;训练阶段,本文将CeLoss和DiceLoss相结合,提高训练效率,解决了正负样本不平衡问题,避免梯度饱和.最终在CamVid数据集上进行了充分的验证实验,结果表明本文算法的mIoU达到约70.18%,而模型参数量约为4.44M,推理速度约为64.1FPS,优于当前主流算法.

Abstract

It is currently a major trend to deploy high-performance neural network algorithms on platforms such as mobile or embedded devices. Aiming at the problem of high complexity of conventional segmentation models, this paper proposes a semantic segmentation algorithm for urban streetscape based on lightweight and multi-scale feature extraction. The algorithm adopts the lightweight network MobileNetV3 as the feature extraction network, introduces the feature pyramid network to embed the shuffle attention in its lateral connection to strengthen the information fusion, then stacks three times the depth-wise asymmetric gating module after its deep features to interact with the local and contextual information; and then employs the multi-scale dilation pyramid module to expand the receptive field to obtain the multi-scale feature information, and finally splices the different levels of information. Finally, the information at different levels is spliced and fused; in the training stage, this paper combines CeLoss and DiceLoss to improve the training efficiency, solve the positive and negative sample imbalance problem, and avoid gradient saturation. Finally, full validation experiments are carried out on CamVid dataset, and the results show that the mIoU of this paper's algorithm reaches about 70.18%, while the number of model parameters is about 4.44M, and the inference speed is about 64.1FPS, which is better than the current mainstream algorithms.

关键词

轻量化网络 / 城市街景 / 语义分割 / 膨胀卷积 / 多尺度特征

Key words

lightweight networks / urban streetscape / semantic segmentation / dilated convolution / multi-scale features

参考文献

[1] Feng D, Haase-Schütz C, Rosenbaum L, et al. Deep multi-modal object detection and semantic segmentation for autonomous driving: datasets, methods, and challenges [J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(03): 1341-1360.

[2] Asgari Taghanaki S, Abhishek K, Cohen J P, et al. Deep semantic segmentation of natural and medical images: a review [J]. Artificial Intelligence Review, 2021, 54: 137-178.

[3] Yuan X, Shi J, Gu L. A review of deep learning methods for semantic segmentation of remote sensing imagery [J]. Expert Systems with Applications, 2021, 169: 114417.

[4] Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation [C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 3431-3440.

[5] Chen L C, Papandreou G, Kokkinos I, et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs [J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 40(04): 834-848.

[6] Poudel R P, Liwicki S, Cipolla R. Fast-scnn: Fast semantic segmentation network [J]. arXiv preprint arXiv: 1902.04502, 2019.

[7] Chen L C, Papandreou G, Schroff F, et al. Rethinking atrous convolution for semantic image segmentation [J]. arXiv preprint arXiv: 1706.05587, 2017.

[8] Zhao H, Shi J, Qi X, et al. Pyramid scene parsing network [C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017: 6230-6239.

[9] Romera E, Álvarez J M, Bergasa L M, et al. ERFNet: Efficient residual factorized convNet for real-time semantic segmentation [J]. IEEE Transactions on Intelligent Transportation Systems, 2018, 19(01): 263-272.

[10] Mehta S, Rastegari M, Caspi A, et al. ESPNet: Efficient spatial pyramid of dilated convolutions for semantic segmentation [M]// Ferrari V, Hebert M, Sminchisescu C, et al. Computer Vision-ECCV 2018: Vol. 11214. Cham: Springer International Publishing, 2018: 561-580.

[11] Zhao H, Qi X, Shen X, et al. Icnet for real-time semantic segmentation on high-resolution images [C]// Proceedings of the European conference on computer vision (ECCV). 2018: 405-420.

[12] Wang Y, Zhou Q, Liu J, et al. Lednet: A lightweight encoder-decoder network for real-time semantic segmentation [C]// 2019 IEEE international conference on image processing (ICIP). IEEE, 2019: 1860-1864.

[13] Li G, Yun I, Kim J, et al. DABNet: Depth-wise asymmetric bottleneck for real-time semantic segmentation [J]. Computer Science, 2019. DOI: 10.48550/arXiv.1907.11357.

[14] Elhassan M A, Yang C, Huang C, et al. S^2-FPN: Scale-aware strip attention guided feature pyramid network for real-time semantic segmentation [J]. arXiv e-prints, 2022: arXiv-2206.

[15] Yu C, Wang J, Peng C, et al. Bisenet: Bilateral segmentation network for real-time semantic segmentation [C]// Proceedings of the European conference on computer vision (ECCV). 2018: 325-341.

[16] Yu C, Gao C, Wang J, et al. BiSeNet V2: Bilateral network with guided aggregation for real-time semantic segmentation [J]. International Journal of Computer Vision, 2021, 129(11): 3051-3068.

[17] Fan M, Lai S, Huang J, et al. Rethinking bisenet for real-time semantic segmentation [C]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 9716-9725.

[18] Hong Y, Pan H, Sun W, et al. Deep dual-resolution networks for real-time and accurate semantic segmentation of road scenes [J]. IEEE Transactions on Intelligent Transportation Systems, 2023(24): 3448-3460.

[19] Milletari F, Navab N, Ahmadi S A. V-net: Fully convolutional neural networks for volumetric medical image segmentation [C]// 2016 fourth international conference on 3D vision (3DV). Ieee, 2016: 565-571.

[20] Zhang Q L, Yang Y B. SA-Net: Shuffle attention for deep convolutional neural networks [Z]. 2021.

[21] Yang Z, Zhu L, Wu Y, et al. Gated channel transformation for visual recognition [C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 11791-11800.

[22] Radosavovic I, Kosaraju R P, Girshick R, et al. Designing network design spaces [C]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 10428-10436.

[23] 赵迪, 孙鹏, 陈奕博, 等. . 基于高度有效驱动注意力与多层级特征融合的城市街景语义分割[J]. 光电子·激光, 2022, 33(10): 1038-1046.

[24] Li H, Xiong P, Fan H, et al. Dfanet: Deep feature aggregation for real-time semantic segmentation [C]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 9522-9531.

[25] Zhou Q, Wang Y, Fan Y, et al. AGLNet: Towards real-time semantic segmentation of self-driving images via attention-guided lightweight network [J]. applied soft computing, 2020, 96: 106682.

[26] Dong G, Yan Y, Shen C, et al. Real-time high-performance semantic image segmentation of urban street scenes [J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 22(06): 3258-3274.

基金资助

国家自然科学基金(62071172)

国家自然科学基金(62202148)