Graph-based spatiotemporal RL framework for sequential task offloading in multi-UAV systems

Efficient collaboration among Uncrewed Aerial Vehicles (UAVs) has significant performance improvement for UAV-based applications. Task offloading is the typical collaboration form for UAV system. However, it still be a challenging problem for UAV system due to task dependencies and the UAV mobility which makes the traditional offloading approaches inefficiency. In this paper, we model the offloading problem as the Sequential Task Offloading Problem (sTOP), which takes the task spatiotemporal dependencies into account. We propose a Graph-based Spatiotemporal Reinforcement Learning (GSTRL) framework, where the environment is modeled as a heterogeneous graph to capture the diverse relationships among system entities. A spatiotemporal state extraction module is designed, which integrates a Heterogeneous Graph Neural Network (HGNN) for spatial dependency modeling and a Long Short-Term Memory (LSTM) network for temporal dynamics. Based on the extracted representations, a masked Proximal Policy Optimization (mPPO) algorithm is proposed to make valid and efficient offloading decisions under multiple system constraints. Extensive experiments using real UAV trajectory and building distribution datasets validate that the proposed method improves the average reward by approximately 25% over state-of-the-art DRL-based and heuristic baselines, by increasing task success rate and operational effectiveness ratio (OER) to 30-50%, while reducing execution time by up to 40% in complex multi-UAV systems.