基于DT-DQN的复杂产品总装物流动态调度

Dynamic Scheduling of Final Assembly Logistics for Complex Products Based on DT-DQN

摘要

复杂产品总装阶段具有物料层级深、齐套依赖强和现场扰动频繁等特征，工位需求变化、AGV故障、通道拥堵与线边库存约束的耦合使得传统静态配送方案难以及时执行。针对中心仓库及线边暂存区至装配工位的物料配送调度问题，本文提出一种基于数字孪生(digital twin，DT)的约束引导深度Q网络(deep Q-network，DQN)动态调度方法(DT-DQN)。建立考虑仓库库存、线边库存、关键物料齐套、车辆容量、车辆可用性和动态通行时间的多周期混合整数线性规划模型，并采用配送任务变量与物料数量变量分离的建模方式，避免多物料同车配送中的通行时间重复计算。数字孪生系统利用执行数据持续校正需求量、库存量、车辆可用状态和通行时间，并以校正后的状态更新下一决策周期的模型参数和可行动作集合。采用工位紧急度排序、候选供给节点筛选、动作掩码和奖励塑形，生成周期内的原子动作序列。以某复杂装备总装车间为背景开展仿真研究，结果表明，与Rolling-MILP相比，DT-DQN在综合扰动场景下加权生产等待时间降低7.1%，方案可行率提高4.9个百分点；与Dispatching Rule相比，物料齐套率提高9.7个百分点。在扩展规模下，DT-DQN表现出较好的在线响应能力，在解质量与约束保持能力之间取得了较好平衡。研究成果可为复杂产品总装物流的动态调度与闭环决策提供理论支撑与方法参考。

Abstract

The final assembly stage of complex products is characterized by deep material hierarchies, strong kitting dependencies, and frequent shop-floor disturbances. Coupled effects of workstation demand changes, automated guided vehicle (AGV) failures, traffic congestion, and line-side inventory constraints make conventional static delivery schedules difficult to execute in a timely manner. To address the material-delivery scheduling problem from the central warehouse and line-side buffers to assembly workstations, this paper proposes a digital twin-based dynamic scheduling method using a constraint-guided deep Q-network (DT-DQN). First, a multi-period mixed-integer linear programming (MILP) model is formulated considering warehouse inventory, line-side inventory, critical material kitting, vehicle capacity, vehicle availability, and dynamic travel time. Delivery task variables are separated from material quantity variables to avoid repeated calculation of travel time when multiple materials are delivered by the same vehicle. Second, the digital twin continuously calibrates material demand, inventory levels, AGV availability, and travel time using real-time execution data. The updated system state is then used to determine the model parameters and feasible action set for the next decision period. Subsequently, atomic action sequences are generated within each decision period through workstation urgency ranking, candidate supply node screening, action masking, and reward shaping. Simulation experiments based on a complex equipment final assembly workshop show that, compared with Rolling-MILP, DT-DQN reduces the weighted production waiting time by 7.1% and improves the solution feasibility rate by 4.9 percentage points under the comprehensive disturbance scenario. Compared with Dispatching Rules, DT-DQN improves the material kitting rate by 9.7 percentage points. Under scaled-up problem instances, DT-DQN exhibits strong online responsiveness, achieving a favorable balance between solution quality and constraint-preservation capability. The proposed method provides theoretical support and practical reference for dynamic scheduling and closed-loop decision-making in complex product final assembly logistics.

关键词

复杂产品总装 / 制造物流 / 数字孪生 / 动态调度 / 约束引导深度强化学习

Key words

complex product final assembly / manufacturing logistics / digital twin / dynamic scheduling / constraint-guided deep reinforcement learning

参考文献

[1] Pan Y H, Qu T, Wu N Q, et al. Digital twin based real-time production logistics synchronization system in a multi-level computing architecture[J]. Journal of Manufacturing Systems, 2021, 58: 246-260.

[2] Guo D Q, Zhong R Y, Lin P, et al. Digital twin-enabled graduation intelligent manufacturing system for fixed-position assembly islands[J]. Robotics and Computer-Integrated Manufacturing, 2020, 63: 101917.

[3] Wang W B, Zhang Y F, Zhong R Y. A proactive material handling method for CPS enabled shop-floor[J]. Robotics and Computer-Integrated Manufacturing, 2020, 61: 101849.

[4] Luo H, Wang K, Kong X T R, et al. Synchronized production and logistics via ubiquitous computing technology[J]. Robotics and Computer-Integrated Manufacturing, 2017, 45: 99-115.

[5] Li M X, Huang G Q. Production-intralogistics synchronization of industry 4.0 flexible assembly lines under graduation intelligent manufacturing system[J]. International Journal of Production Economics, 2021, 241: 108272.

[6] Wang S Y, Huang G Q. Meta-inventory[J]. Robotics and Computer-Integrated Manufacturing, 2023, 81: 102503.

[7] 林国义, 郭慧妍, 冷杰武, 等 . 数字孪生在工业工程领域应用的热点和趋势分析[J]. 工业工程, 2024, 27(6): 13-25.

[8] Lin Guoyi, Guo Huiyan, Leng Jiewu, et al. Hotspot and trend analysis of digital twin application in industrial engineering[J]. Industrial Engineering Journal, 2024, 27(6): 13-25.

[9] Zhu Y H, Cheng J F, Liu Z F, et al. Production logistics digital twins: Research profiling, application, challenges and opportunities[J]. Robotics and Computer-Integrated Manufacturing, 2023, 84: 102592.

[10] Ouahabi N, Chebak A, Kamach O, et al. Leveraging digital twin into dynamic production scheduling: a review[J]. Robotics and Computer-Integrated Manufacturing, 2024, 89: 102778.

[11] Tao F, Zhang M. Digital twin shop-floor: A new shop-floor paradigm towards smart manufacturing[J]. IEEE Access, 2017, 5: 20418-20427.

[12] Zhuang C B, Liu J H, Xiong H. Digital twin-based smart production management and control framework for the complex product assembly shop-floor[J]. The International Journal of Advanced Manufacturing Technology, 2018, 96(1): 1149-1163.

[13] Kerrouchi S, Aghezzaf E H, Cottyn J. Production digital twin: A systematic literature review of challenges[J]. International Journal of Computer Integrated Manufacturing, 2024, 37(10/11): 1168-1193.

[14] Ferrari A, Mangano G, Zenezini G. Digital twin applications for intralogistics processes: a literature review[J]. IFAC-PapersOnLine, 2025, 59(10): 1630-1635.

[15] Qi Q L, Tao F, Hu T L, et al. Enabling technologies and tools for digital twin[J]. Journal of Manufacturing Systems, 2021, 58: 3-21.

[16] Yi Y, Yan Y H, Liu X J, et al. Digital twin-based smart assembly process design and application framework for complex products and its case study[J]. Journal of Manufacturing Systems, 2021, 58: 94-107.

[17] Dauzère-Pérès S, Ding J W, Shen L J, et al. The flexible job shop scheduling problem: a review[J]. European Journal of Operational Research, 2024, 314(2): 409-432.

[18] Zhang C, Juraschek M, Herrmann C. Deep reinforcement learning-based dynamic scheduling for resilient and sustainable manufacturing: a systematic review[J]. Journal of Manufacturing Systems, 2024, 77: 962-989.

[19] Ren F J, Liu H B. Dynamic scheduling for flexible job shop based on MachineRank algorithm and reinforcement learning[J]. Scientific Reports, 2024, 14: 29741.

[20] Cheng W Y, Zhang C Y, Meng L L, et al. A cooperative agent deep reinforcement learning framework for solving flexible job shop scheduling problem with automated guided vehicles[J]. Expert Systems with Applications, 2025, 287: 128142.

[21] Gui Y, Tang D B, Zhu H H, et al. Dynamic scheduling for flexible job shop using a deep reinforcement learning approach[J]. Computers & Industrial Engineering, 2023, 180: 109255.

[22] Zhang M, Wang L, Qiu F S, et al. Dynamic scheduling for flexible job shop with insufficient transportation resources via graph neural network and deep reinforcement learning [J]. Computers & Industrial Engineering, 2023, 186: 109718.

[23] Jaoua A, Masmoudi S, Negri E. Digital twin-based reinforcement learning framework: application to autonomous mobile robot dispatching[J]. International Journal of Computer Integrated Manufacturing, 2024, 37(10/11): 1335-1358.

[24] Lee D, Kang Y S, Do Noh S. Digital twin-driven deep reinforcement learning for real-time optimisation in dynamic AGV systems[J]. International Journal of Production Research, 2026, 64(1): 106-124.

[25] Zhang L X, Yang C, Yan Y, et al. Automated guided vehicle dispatching and routing integration via digital twin with deep reinforcement learning[J]. Journal of Manufacturing Systems, 2024, 72: 492-503.

[26] Xu Z G, Song H L, Yang S L, et al. Dynamic scheduling for an assembly workshop with insufficient AGVs using deep reinforcement learning[J]. Journal of Mechanical Science and Technology, 2025, 39(8): 4625-4637.

[27] Chang X, Jia X L, Fu S F, et al. Digital twin and deep reinforcement learning enabled real-time scheduling for complex product flexible shop-floor[J]. Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture, 2023, 237(8): 1254-1268.

[28] 祝正宇, 郭具涛, 吕佑龙, 等 . 面向柔性作业车间生产调度的深度强化学习方法[J]. 中国机械工程, 2024, 35(11): 2007-2014,2034.

[29] Zhu Zhengyu, Guo Jutao, Lyu Youlong, et al. Deep reinforcement learning method for flexible job shop scheduling[J]. China Mechanical Engineering, 2024, 35(11): 2007-2014,2034.

[30] 李子晨, 苑明海, 黄涵钰, 等 . 基于深度强化学习的作业车间节能调度研究[J]. 制造技术与机床, 2024(6): 161-169.

[31] Li Zichen, Yuan Minghai, Huang Hanyu, et al. Research on energy-saving scheduling of job shop based on deep reinforcement learning[J]. Manufacturing Technology & Machine Tool, 2024(6): 161-169.

[32] 罗梓珲, 江呈羚, 刘亮, 等 . 基于深度强化学习的智能车间调度方法研究[J]. 物联网学报, 2022, 6(1): 53-64.

[33] Luo Zihui, Jiang Chengling, Liu Liang, et al. Research on deep reinforcement learning based intelligent shop scheduling method[J]. Chinese Journal on Internet of Things, 2022, 6(1): 53-64.

[34] 张颖伟, 高鸿瑞, 张鼎森, 等 . 基于多智能体的数字孪生及其在工业中应用的综述[J]. 控制与决策, 2023(8): 2168-2182.

[35] Zhang Yingwei, Gao Hongrui, Zhang Dingsen, et al. A review of multi-agent-based digital twins and its application in industry[J]. Control and Decision, 2023(8): 2168-2182.

[36] 柳虎威, 梁凯博, 杨江龙, 等 . 数字孪生驱动的城市应急配送动态优化[J]. 工业工程, 2026, 29(1): 61-74.

[37] Liu Huwei, Liang Kaibo, Yang Jianglong, et al. Digital twin-driven dynamic optimization for urban emergency distribution[J]. Industrial Engineering Journal, 2026, 29(1): 61-74.

[38] Liu W R, Zou X F, Wen Z W, et al. Digital twin-based production-logistics synchronization system for satellite mass assembly shop-floor[J]. Chinese Journal of Mechanical Engineering, 2024, 37(1): 163.

[39] Rizqi Z U, Chou S Y, Cahyo W N. A simulation-based digital twin for smart warehouse: towards standardization[J]. Decision Analytics Journal, 2024, 12: 100509.

基金资助

国家社会科学基金项目(24FGLB047)

北京经理学院科技英才支持计划(25KJYCLH01)

北京经理学院科技创新团队建设项目(24KJTD01)