Blade Stitching Technology Based on Image Semantic Segmentation and Structure Preservation Constraints

doi:10.3969/j.issn.1009-086x.2024.04.013

Abstract

Abstract:

Taking UAV's patrol shooting of fan blades as the application background， the image stitching method of fan blade images is studied， a method of background segmentation and image gridding stitching is proposed. Through in-depth learning of U-Net algorithm， the main part of the fan is extracted from the image， which can effectively deal with the difficult problem of fan blade splicing caused by large parallax and uneven distribution of target background feature points. In the process of image grid optimization， the energy function is designed based on the strategy of protecting the global linearity of the fan blade， and the grid vertex is optimized to the edge of the fan. Linear features are effectively protected. The natural stitching of several fan blades is realized. The stitching image has small distortion and continuous reality.

Key words: image semantics segmentation, as-projective-as-possible （APAP）, meshed image stitching, line structure preservation, deep learning, feature extraction

摘要：

以无人机对风机叶片的巡检拍摄为应用背景，开展了风机叶片图像的拼接方法研究，提出了一种先进行背景分割然后进行图像网格化拼接的处理方法。通过深度学习U-Net算法，进行图像中风机主体部分的提取，该处理能够有效处理多视角大视差、目标背景特征点分布不均导致的风机叶片拼接困难的问题；在图像网格优化的过程中，基于保护风叶全局线性度的策略设计能量函数，优化得到的网格顶点对风机边缘的直线特征进行了有效保护。实现了多幅风机叶片的自然拼接，拼接得到的图像视觉效果畸变小、连续真实。

关键词: 图像语义分割, as-projective-as-possible（APAP）, 网格化图像拼接, 直线结构保持, 深度学习, 特征提取

CLC Number:

TP391.41

Henglei XU, Shuaiqi CHEN, Xun SONG, Mingjie ZHU. Blade Stitching Technology Based on Image Semantic Segmentation and Structure Preservation Constraints[J]. Modern Defense Technology, 2024, 52(4): 123-129.

许恒雷, 陈帅旗, 宋勋, 朱洺洁. 基于图像语义分割与结构保持约束的风机叶片拼接技术[J]. 现代防御技术, 2024, 52(4): 123-129.

Figures/Tables 10

References 13

1	韩斌，赵勇，邓巍，等. 无人机近距离巡检风电机组叶片应用研究［J］. 风能， 2017（9）： 46-52.
	HAN Bin， ZHAO Yong， DENG Wei， et al. Application Research of UAV Close-Range Inspection of Wind Turbine Blades［J］. Wind Energy， 2017（9）： 46-52.
2	ZARAGOZA J， CHIN T J， TRAN Q H， et al. As-Projective-As-Possible Image Stitching with Moving DLT［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2014， 36（7）： 1285-1298.
3	IGARASHI T， IGARASHI Y. Implementing As-Rigid-As-Possible Shape Manipulation and Surface Flattening［J］. Journal of Graphics， GPU， and Game Tools， 2009， 14（1）： 17-30.
4	JOO K， KIM N， OH T H， et al. Line Meets As-Projective-As-Possible Image Stitching with Moving DLT［C］∥2015 IEEE International Conference on Image Processing （ICIP）. Piscataway： IEEE， 2015： 1175-1179.
5	LIN C C， PANKANTI S U， RAMAMURTHY K N， et al. Adaptive As-Natural-As-Possible Image Stitching［C］∥2015 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE， 2015： 1155-1163.
6	CHEN Yusheng， CHUANG Y Y. Natural Image Stitching with the Global Similarity Prior［C］∥Computer Vision-ECCV 2016. Cham： Springer International Publishing， 2016： 186-201.
7	RONNEBERGER O， FISCHER P， BROX T. U-Net： Convolutional Networks for Biomedical Image Segmentation［C］∥Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015. Cham： Springer International Publishing， 2015： 234-241.
8	ZHOU Zongwei， RAHMAN SIDDIQUEE M M， TAJBAKHSH N， et al. UNet++： A Nested U-Net Architecture for Medical Image Segmentation［M］∥STOYANOV D， TAYLOR Z， CARNEIRO G， et al. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Cham： Springer International Publishing， 2018： 3-11.
9	HUANG Huimin， LIN Lanfen， TONG Ruofeng， et al. UNet 3+： A Full-Scale Connected UNet for Medical Image Segmentation［C］∥ICASSP 2020—2020 IEEE International Conference on Acoustics， Speech and Signal Processing （ICASSP）. Piscataway： IEEE， 2020： 1055-1059.
10	HE Kaiming， ZHANG Xiangyu， REN Shaoqing， et al. Deep Residual Learning for Image Recognition［C］∥2016 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE， 2016： 770-778.
11	IBTEHAZ N， RAHMAN M S. MultiResUNet： Rethinking the U-Net Architecture for Multimodal Biomedical Image Segmentation［J］. Neural Networks， 2020， 121： 74-87.
12	陈立伟，宋涛羽. 基于语义分割的图像拼接实验设计［J］. 实验技术与管理， 2021， 38（12）： 48-51， 56.
	CHEN Liwei， SONG Taoyu. Experimental Design of Image Mosaic Based on Semantic Segmentation［J］. Experimental Technology and Management， 2021， 38（12）： 48-51， 56.
13	GROMPONE VON GIOI R， JAKUBOWICZ J， MOREL J M， et al. LSD： A Fast Line Segment Detector with a False Detection Control［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2010， 32（4）： 722-732.

图像编号	原始图像特征对数量	U-Net分割后特征对数量	图像编号	原始图像特征对数量	U-Net分割后特征对数量
1-2	159	15	2-3	159	43
3-4	159	50	4-5	159	48
5-6	159	54	6-7	159	79
7-8	159	107	8-9	159	75
9-10	159	58	10-11	863	64
11-12	750	85	12-13	778	88
13-14	881	101	14-15	859	78
15-16	108	13	16-17	585	73
17-18	614	69	18-19	406	121
19-20	69	131	20-21	245	132
21-22	161	24	22-23	214	83

图像编号	原始图像特征对数量	U-Net分割后特征对数量	图像编号	原始图像特征对数量	U-Net分割后特征对数量
1-2	159	15	2-3	159	43
3-4	159	50	4-5	159	48
5-6	159	54	6-7	159	79
7-8	159	107	8-9	159	75
9-10	159	58	10-11	863	64
11-12	750	85	12-13	778	88
13-14	881	101	14-15	859	78
15-16	108	13	16-17	585	73
17-18	614	69	18-19	406	121
19-20	69	131	20-21	245	132
21-22	161	24	22-23	214	83

试验情况	文本算法（SIFT+结构保持）	SIFT特征	KAZE	BRISK	ORB
总叶片数量（组）	22	22	22	22	22
图片总数（张）	354	354	354	354	354
拼接成功数（组）	20	14	9	6	7
拼接失败数（组）	2	8	13	16	15
拼接成功率/%	91	63	41	27	32

试验情况	文本算法（SIFT+结构保持）	SIFT特征	KAZE	BRISK	ORB
总叶片数量（组）	22	22	22	22	22
图片总数（张）	354	354	354	354	354
拼接成功数（组）	20	14	9	6	7
拼接失败数（组）	2	8	13	16	15
拼接成功率/%	91	63	41	27	32

[1]	Yongjun TU, Hongsheng LIN, Zhiyong WANG. Underwater Image Background and Foreground Segmentation Method with Parallel Convolution [J]. Modern Defense Technology, 2024, 52(3): 104-111.
[2]	Yang LI, Xiao LIANG, Haisa JIA. Ship Detection in SAR Images Based on Knowledge Distillation [J]. Modern Defense Technology, 2023, 51(4): 78-85.
[3]	Baoyin YAO, Lei MAO, Zhibin WANG. Applications of Artificial Intelligence in Space Equipment [J]. Modern Defense Technology, 2023, 51(2): 33-42.
[4]	Lei YUE, Jianhu YUAN, Liu YANG, Tingting LÜ. Research on a Patrol Duty Object Detection Algorithm [J]. Modern Defense Technology, 2023, 51(1): 67-74.
[5]	Jia-wei BAO, Xiao-ping TIAN, Yu-na LIU, Chang-kuan ZOU, Yu-qing ZHANG. Summary of Research on Security Issues of ADS-B Protocol [J]. Modern Defense Technology, 2022, 50(5): 28-35.
[6]	YAO Bao-yin, MAO Lei, XIAO Ke, QU Hui. Applications of Artificial Intelligence on the Optical Earth Observation [J]. Modern Defense Technology, 2021, 49(5): 26-31.
[7]	WANG Wen-qing, PANG Ying, LIU Yang, YANG Dong-fang, ZHANG Meng. Air-to-Ground Target Intelligent Detection Algorithm Based on Dual Attention Mechanism [J]. Modern Defense Technology, 2020, 48(6): 81-88.
[8]	WANG Zhi-hu, SHEN Xiao-qing, GUI Wei-long. Survey of Small Targets Detection Technology in Optical Imaging [J]. Modern Defense Technology, 2020, 48(5): 67-73.
[9]	TANG Run-ze, ZHANG Cheng-long, LI Lin-lin. Application of Artificial Intelligence on Situation Assessment and Game Countermeasure in Unmanned Battlefield [J]. Modern Defense Technology, 2020, 48(5): 25-31.
[10]	LIANG Fu-tai, LI Hong-quan, MENG Qing-wen, CAI Li. An Aerial Target Track Clustering Method [J]. Modern Defense Technology, 2020, 48(1): 19-25.
[11]	MA Xin-xing, TENG Ke-nan, HOU Xue-long. Forecast Model of Main Air Attack Direction Based on CNN [J]. Modern Defense Technology, 2018, 46(5): 6-12.