基于哈希图的数据库信息批量无损迁移方法

Batch Lossless Migration Method for Database Information Based on Hash Graph

摘要

针对在数据迁移过程中, 为保证数据的一致性和完整性, 避免数据丢失或损坏, 降低重复数据量的问题, 提出了一种基于哈希图的数据库信息批量无损迁移方法。根据数据库数据属性的相似度和二分图最优匹配算法获得数据库数据间的相似度, 并根据相似度完成重复数据的检测和删除。通过局部敏感哈希函数计算去重后各数据的哈希值, 基于所有数据的哈希值建立哈希图。利用条件深度卷积生成对抗网络对数据进行批量迁移, 并对迁移后的数据哈希值和哈希图中对应的值进行比较, 对哈希值不同的数据重新实施迁移, 以确保数据库信息的无损迁移。实验结果表明, 该方法的信息迁移速度较快, 数据重复率较低, 并且能保证数据的一致性, 说明其具备可行性。

Abstract

In the process of data migration, it is necessary to ensure the consistency and integrity of data, avoid data loss or damage, and reduce the amount of duplicate data. Therefore, a hash graph based batch lossless database information migration method is proposed. The similarity of data between database is obtained based on the similarity of database data attributes and the optimal matching algorithm for bipartite graphs, the detection and deletion of duplicate data is completed based on the similarity. The hash values of each data after deduplication is calculated using a locally sensitive hash function, and a hash graph based on the hash values of all data is established. Conditional deep convolution is utitized to generate adversarial networks for batch migration of data, and the hash values of the migrated data is compared with the corresponding values in the hash graph. The migration is re-expanding for data with different hash values to ensure lossless migration of database information. The experimental results show that this method has a fast information transfer speed, low data duplication rate, and can ensure data consistency, indicating that it has achieved the research objectives and is feasible.

关键词

哈希图 / 数据库信息 / 无损迁移 / 局部敏感哈希函数 / 条件深度卷积生成对抗网络

Key words

Hash graph / database information / non destructive migration / locally sensitive hash function / conditional deep convolution generates adversarial networks

参考文献

[1] 陈金浩, 蒋大鹏, 张怡卓, 等. 实木板材抗弯强度的SWCSS-GFK-SVM数据迁移建模方法[J]. 光谱学与光谱分析, 2022, 42(5): 1471-1477.

[2] CHEN J H, JIANG D P, ZHANG Y Z, et al. Research on Data Migration Modeling Method for Bending Strength of Solid Wood Based on SWCSS-GFK-SVM[J]. Spectroscopy and Spectral Analysis, 2022, 42(5): 1471-1477.

[3] 王甫棣, 王帅, 赵希鹏. 基于ElasticSearch的异构数据迁移方法[J]. 气象科技, 2022, 50(1): 45-50.

[4] WANG F D, WANG S, ZHAO X P. A Method for Heterogeneous Data Migration Based on Elastic Search[J]. Meteorological Science and Technology, 2022, 50(1): 45-50.

[5] 彭飞, 贲驰, 马煜, 等. 用于短期风功率预测的历史数据深度迁移模型[J]. 重庆大学学报, 2022, 45(1): 95-102.

[6] PENG F, BEN C, MA Y, et al. A Short-Term Wind Power Prediction Model Based on Deep Transfer Learning of Historical Data[J]. Journal of Chongqing University, 2022, 45(1): 95-102.

[7] 陈洪健, 季健, 洪帅, 等. 基于ClickHouse的版本化数据迁移方法[J]. 计算机应用, 2022, 42(S2): 105-110.

[8] CHEN H J, JI J, HONG S, et al. Versioned Data Migration Method Based on ClickHouse[J]. Journal of Computer Applications, 2022, 42(S2): 105-110.

[9] LOTFOLLAHI M, NAGHIPOURFAR M, LUECKEN M D, et al. Mapping Single-Cell Data to Reference Atlases by Transfer Learning[J]. Nature Biotechnology, 2022, 40(1): 121-130.

[10] 吴宪, 汤红波, 赵宇, 等. 一种有状态容器跨集群实时迁移方法[J]. 计算机研究与发展, 2024, 61(2): 494-502.

[11] WU X, TANG H B, ZHAO Y, et al. A Real-Time Migration Method for Stateful Containers across Clusters[J]. Computer Research and Development, 2024, 61(2): 494-502.

[12] 朱海洋, 钱中昊, 严凡, 等. 支持多维度数据去重的交互式可视分析方法[J]. 计算机辅助设计与图形学学报, 2022, 34(6): 841-851.

[13] ZHU H Y, QIAN Z H, YAN F, et al. An Interactive Visual Analysis Method for Multi-Dimensional Data Deduplication[J]. Journal of Computer-Aided Design & Computer Graphics, 2022, 34(6): 841-851.

[14] 童晓阳, 权文杰, 李智, 等. 利用行波波形改进编辑距离的柔性直流输电线路后备保护方案[J]. 电网技术, 2023, 47(1): 294-300.

[15] TONG X Y, QUAN W J, LI Z, et al. Backup Protection Scheme for Flexible DC Transmission Line by Traveling Wave Waveform Similarity Based on Improved Editing Distance[J]. Power System Technology, 2023, 47(1): 294-300.

[16] 詹康, 王逸文, 何熊熊. 基于数据相似度和引力理论的密度峰聚类算法[J]. 高技术通讯, 2023, 33(1): 88-96.

[17] ZHAN K, WANG Y W, HE X X. Density Peak Clustering Algorithm Based on Data Similarity and Gravity Theory[J]. Chinese High Technology Letters, 2023, 33(1): 88-96.

[18] 林伟杰, 王勇, 周林. 基于加权二分图的K均值最佳聚类数确定算法[J]. 计算机工程与设计, 2023, 44(4): 1104-1111.

[19] LIN W J, WANG Y, ZHOU L. Deter Mination Algorithm of K-Means Optimal Clustering Number Based on Weighted Bipartite Graph[J]. Computer Engineering and Design, 2023, 44(4): 1104-1111.

[20] 张仕, 赖会霞, 肖如良, 等. 开放环境多分布特性的局部敏感哈希检索方法[J]. 软件学报, 2022, 33(4): 1200-1217.

[21] ZHANG S, LAI H X, XIAO R L, et al. Open Environmental Locality-Sensitive Hashing Retrieval for Multiple Distributed Characteristics[J]. Journal of Software, 2022, 33(4): 1200-1217.

[22] 方伟, 王玉, 闫文君, 等. 基于DTW的复杂飞行动作快速评估方法研究[J]. 中国电子科学研究院学报, 2022, 17(1): 44-49.

[23] FANG W, WANG Y, YAN W J, et al. Research on the Rapid Evaluation Method of Complex Flight Action Based on DTW[J]. Journal of China Academy of Electronics and Information Technology, 2022, 17(1): 44-49.

[24] 赵莉, 王莎莎, 刘道华, 等. 一种改进感知哈希算法的2DPCANet人脸识别方法[J]. 信阳师范学院学报(自然科学版), 2022, 35(3): 494-498.

[25] ZHAO L, WANG S S, LIU D H, et al. A 2DPCANet Face Recognition Method Based on Improved Perceptual Hashing Algorithm[J]. Journal of Xinyang Normal University (Natural Science Edition), 2022, 35(3): 494-498.

[26] 张浩, 齐光磊, 侯小刚, 等. 基于改进Fisher准则的深度卷积生成对抗网络算法[J]. 光学精密工程, 2022, 30(24): 3239-3249.

[27] ZHANG H, QI G L, HOU X G, et al. Deep Convolutional Generative Adversarial Network Algorithm Based on Improved Fisher's Criterion[J]. Optics and Precision Engineering, 2022, 30(24): 3239-3249.

基金资助

河北省教育科学“十四五”规划基金资助项目(2303150)