Automating and scaling tightly-coupled multiscale models

  • Date:

    July 11

  • Speaker:

    Ivan Kondov (RTG 2450, KIT, Germany)

  • Time:

    14:35 - 14:55

  • Tightly coupled task-based multiscale models do not scale when implemented using a traditional workflow management system. This is because the fine-grained task parallelism of such applications cannot be exploited efficiently due to scheduling and communication overheads. Existing tools and frameworks allow implementing efficient task-level parallelism, however with high programming effort. On the other hand, Dask and Parsl are Python libraries for low-effort up-scaling of task-parallel applications but still require considerable programming effort and do not equally provide functions for optimal task scheduling. By extending the wfGenes tool with new generators and a static task graph scheduler, we enhance Dask and Parsl to tackle these deficiencies and to generate optimized input for these systems from a simple application description and enable rapid design of scalable task-parallel multi- scale applications relying on thorough graph analysis and automatic code generation. The performance of the generated code has been analyzed by using random task graphs with up to 10,000 nodes and executed on thousands of CPU cores. The approach implemented in wfGenes is beneficial for improving the usability and increasing the exploitation of existing tools, and for increasing productivity of multiscale modeling scientists.