We use cookies to improve your experience with our site.
Zhi-Feng Yu, Wei-Song Shi. Queue Waiting Time Aware Dynamic Workflow Scheduling in Multicluster Environments[J]. Journal of Computer Science and Technology, 2010, 25(4): 864-873. DOI: 10.1007/s11390-010-1067-6
Citation: Zhi-Feng Yu, Wei-Song Shi. Queue Waiting Time Aware Dynamic Workflow Scheduling in Multicluster Environments[J]. Journal of Computer Science and Technology, 2010, 25(4): 864-873. DOI: 10.1007/s11390-010-1067-6

Queue Waiting Time Aware Dynamic Workflow Scheduling in Multicluster Environments

  • Workflows are prevailing in scientific computation. Multicluster environments emerge and provide more resources, benefiting workflows but also challenging the traditional workflow scheduling heuristics. In a multicluster environment, each cluster has its own independent workload management system. Jobs are queued up before getting executed, they experience different resource availability and wait time if dispatched to different clusters. However, existing scheduling heuristics neither consider the queue wait time nor balance the performance gain with data movement cost. The proposed algorithm leverages the advancement of queue wait time prediction techniques and empirically studies if the tunability of resource requirements helps scheduling. The extensive experiment with both real workload traces and test bench shows that the queue wait time aware algorithm improves workflow performance by 3 to 10 times in terms of average makespan with relatively very low cost of data movement.
  • loading

Catalog

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return