为高性能计算系统构建一种程序错误弹性感知的检查点机制
ResCheckpointer: Building Program Error Resilience-Aware Checkpointing Mechanism for HPC Systems
-
摘要:研究背景 高性能计算系统广泛应用于解决科学和工程领域的复杂问题,助力行业发展和技术进步。在摩尔定律的驱动下,高性能计算系统的计算能力不断提升。然而,随着硬件尺寸缩小、电路密度增加以及近阈值操作等硬件技术的应用,系统的平均故障间隔时间仅为数小时。这给需要长时间运行的计算任务(如天气模拟等)带来了稳定性挑战。这些任务通常需要数天、数周、甚至数月才能完成,故障的发生会中断计算过程,导致大量的时间和资源浪费。检查点恢复机制可以帮助降低系统故障带来的风险。通过周期性地保存程序运行的中间状态(即检查点),检查点恢复机制可以在程序崩溃时通过加载最近保存的检查点来恢复程序的正常运行,避免程序从头开始执行完整程序。目的 引入检查点恢复机制导致高性能计算系统需要承担额外的计算和存储资源开销,而系统故障率的不断上升使得这些开销愈加昂贵。因此,优化检查点恢复机制的开销是提高系统性能的关键。本文构建基于图神经网络的程序错误弹性预测模型Crash-Predictor以及程序错误弹性感知的检查点机制ResCheckpointer,来灵活调整检查点间隔,以优化系统检查点恢复机制的开销。方法 本文首先构建基于图神经网络的程序错误弹性感知学习范式,提出Crash-Predictor用于有效预测程序的崩溃倾向性。随后,基于Crash-Predictor构建程序错误弹性感知的检查点机制ResCheckpointer,以根据预测的程序错误弹性来灵活地调整检查点间隔。本文选择10个来自不同领域的程序,构建相应的程序错误弹性图,以训练Crash-Predictor并评估其精度,然后根据其所预测的程序错误弹性调整检查点间隔,验证ResCheckpointer的有效性。结果 Crash-Predictor的平均绝对损失在所有基准测试程序中的范围为0.016~0.067。该结果表明本文构建的程序错误弹性图能够有效表征程序错误弹性知识,并且所设计的图神经网络学习范式能够有效学习程序错误弹性。此外,Crash-Predictor比现有基于传统机器学习模型的方法具有更小的平均误差,分别比PARIS和G-SEPM少0.0082和0.0288。和基准检查点恢复机制相比,ResCheckpointer最多可降低55.37%的检查点恢复机制带来的开销。从结果中还可以发现,ResCheckpointer在具有较强错误弹性的高性能计算程序上的效果更好。通过调整平均故障间隔时间,发现对于较大的平均故障间隔时间,ResCheckpointer根据程序执行阶段对检查点间隔进行更精细的调整,比根据程序的整体崩溃倾向调整检查点的性能更好。此外,Res-Checkpointer表现得比根据程序平均崩溃倾向性调整间隔的AVGI方法更好,说明利用Crash-Predictor进行较为准确的错误弹性预测能够有效地为检查点间隔设置提供指导。结论 实验结果验证了本文提出的基于图神经网络的程序弹性预测模型Crash-Predictor能够有效地为检查点间隔优化提供指导,根据程序和程序不同执行阶段的错误弹性调整检查点间隔的检查点机制ResCheckpointer能够有效地降低系统引入检查点恢复机制所需的开销。未来,ResCheckpointer可推广到多任务情景,综合考虑多个程序的错误、弹性优化检查点恢复机制开销。此外,还可以结合启发式修复方法对崩溃程序进行修复,增强程序的错误弹性,更进一步优化检查点恢复机制。Abstract: The reliability of high-performance computing (HPC) is essential for program execution stability. However, as the hardware fault rate constantly increases, fault-tolerance techniques such as Checkpoint/Restart (C/R) introduce significant system overhead. This paper proposes Program Error Resilience-Aware Checkpointing Mechanism (ResCheckpointer) to mitigate the overhead of the C/R mechanism. The primary motivation of ResCheckpointer is that we observe that crash proneness (i.e., the probability of the program crashing after fault occurrence) varies significantly among inter- and intra-HPC programs, which prompts us to flexibly adjust checkpoint intervals for further C/R overhead optimization. Specifically, we first construct the graph neural network (GNN) based learning paradigms to excavate the complex error propagation and effect mechanisms hidden within the HPC program’s execution flow, and propose Crash-Predictor for efficiently predicting programs’ crash proneness. Based on this, we build ResCheckpointer, which equips an intelligent checkpoint interval setting strategy for HPC programs, i.e., denser for the crash proneness stage while sparser for the error resilience stage. Experimental results show that ResCheckpointer can achieve up to 55.37% C/R cost reduction compared with the baseline C/R mechanism.