RIME-Transformer模型在复杂时序预测问题中的应用

Application of RIME-Transformer model in complex time series prediction problems

摘要

为解决传统Transformer在长序列建模和计算效率的不足,本文提改进的Transformer模型,该模型在特征提取阶段引入多尺度卷积结构,通过并行卷积核在不同尺度上捕捉短期波动与长期趋势,增强对多层次时序模式的表征能力。随后,模型采用可学习的位置编码代替固定的正弦编码,更好地应对非平稳数据和不规则时间间隔问题。在全局依赖建模过程中,改进的编码器利用多头自注意力机制建立跨时间步的特征交互,动态分配时刻权重以聚焦关键片段,有效降低长序列建模的计算复杂度。Transformer模型还结合了霜冰优化算法(rime optimization algorithm, RIME)在高维超参数空间中进行高效搜索与优化,提升模型的收敛速度与泛化能力。实验在3个真实复杂数据集上进行,结果表明RIME-Transformer模型在多项指标上均优于主流方法研究结果,验证所提模型在复杂时序预测任务中的有效性与优越性。

Abstract

To address the shortcomings of the traditional Transformer in long-term sequence modeling and computational efficiency, an improved Transformer model is proposed. This model first introduces a multi-scale convolutional structure in the feature extraction stage. Parallel convolution kernels capture both short-term fluctuations and long-term trends at different scales, thereby enhancing the representation of multi-level temporal patterns. Subsequently, the model employs learnable positional encoding instead of fixed sinusoidal encoding to better address the challenges posed by non-stationary data and irregular time intervals. During global dependency modeling, the improved encoder leverages a multi-head self-attention mechanism to establish feature interactions across time steps and dynamically assign moment weights to focus on key segments, effectively reducing the computational complexity of long-term sequence modeling. Furthermore, the model incorporates the rime optimization algorithm RIME for efficient search and optimization in a high-dimensional hyperparameter space, thereby improving the model's convergence speed and generalization ability. Experiments on three real-world complex datasets demonstrate that the RIME-Transformer outperforms mainstream methods across multiple metrics. These results validate the effectiveness and superiority of the proposed model for complex time series prediction tasks.

关键词

Transformer / 霜冰优化算法 / 多尺度特征编码 / 可学习位置编码 / 注意力池化

Key words

Transformer / rime optimization algorithm / multi-scale feature encoding / learnable positional encoding / attention pooling

参考文献

[1] LIM B , ZOHREN S . Time—series forecasting with deep learning: a survey[J]. Philosophical Transactions of the Royal Society A, 2021, 379(2194): 20200209.

[2] TORRES J F , HADJOUT D , SEBAA A , et al. Deep learning for time series forecasting: a survey[J]. Big Data, 2021, 9(1): 3-21.

[3] MASINI R P , MEDEIROS M C , MENDES E F . Machine learning advances for time series forecasting[J]. Journal of Economic Surveys, 2023, 37(1): 76-111.

[4] WANG Zihan , KONG Fanheng , FENG Shi , et al. Is mamba effective for time series forecasting[J]. Neurocomputing, 2025, 619: 129178.

[5] BHOGADE V , NITHYA B . Time series forecasting using Transformer neural network[J]. International Journal of Computers and Applications, 2024, 46(10): 880-888.

[6] KIM D K , KIM K . A convolutional Transformer model for multivariate time series prediction[J]. IEEE Access, 2022, 10: 101319-101329.

[7] KHAN S , NASEER M , HAYAT M , et al. Transformers in vision: a survey[J]. ACM Computing Surveys (CSUR), 2022, 54(10s): 1-41.

[8] SUNKI A , SATYAKUMAR C , NARAYANA G S , et al. Time series forecasting of stock market using ARIMA, LSTM and FBprophet[C]// MATEC Web of Conferences. [S.l]: EDP Sciences, 2024, 392: 01163.

[9] TOKGÖZ A , ÜNAL G . A RNN based time series approach for forecasting Turkish electricity load[C]// 2018 26th Signal Processing and Communications Applications Conference (SIU). Izmir: IEEE, 2018: 1-4.

[10] SAGHEER A , KOTB M . Time series forecasting of petroleum production using deep LSTM recurrent networks[J]. Neurocomputing, 2019, 323: 203-213.

[11] KIM J , MOON N . BiLSTM model based on multivariate time series data in multiple field for forecasting trading area[J]. Journal of Ambient Intelligence and Humanized Computing, 2020, 11(2): 599-608.

[12] VASWANI A , SHAZEER N , PARMAR N , et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30: 5998-6008.

[13] ZHOU Haoyi , ZHANG Shanghang , PENG Jieqi , et al. Informer: beyond efficient Transformer for long sequence time—series forecasting[C]// Proceedings of the AAAI Conference on Artificial Intelligence. Philadelphia: AAAI, 2021, 35(12): 11106-11115.

[14] WU Haixu , XU Jiehui , WANG Jianmin , et al. Autoformer: decomposition Transformers with auto—correlation for long—term series forecasting[J]. Advances in Neural Information Processing Systems, 2021, 34: 22419-22430.

[15] AHAMED M A , CHENG Qiang . Time machine: a time series is worth 4 mambas for long—term forecasting[C]// ECAI 2024: 27th European Conference on Artificial Intelligence. Santiago de Compostela: IOS Press, 2024, 392: 1688.

[16] LIN Shengsheng , LIN Weiwei , WU Wentai , et al. Petformer: long—term time series forecasting via placeholder—enhanced Transformer[J]. IEEE Transactions on Emerging Topics in Computational Intelligence, 2024, 9(2): 1189-1201.

[17] SU Hang , ZHAO Dong , HEIDARI ALI ASGHAR , et al. RIME: a physics—based optimization[J]. Neurocomputing, 2023, 532: 183-214.

[18] CAMBRIA E , WHITE B . Jumping NLP curves: a review of natural language processing research[J]. IEEE Computational Intelligence Magazine, 2014, 9(2): 48-57.

[19] ZHONG Junliu , PUN Chi—man . An end—to—end dense—inception net for image copy—move forgery detection[J]. IEEE Transactions on Information Forensics and Security, 2019, 15: 2134-2146.

基金资助

国家自然科学基金资助项目(61806001)