Talks

Deep neural networks (DNNs) exhibit excellent performance for many machine learning tasks, e.g., image classification, natural language processing, and game playing. However, training DNNs remains challenging and computationally expensive, with much room for improvement, both in terms of new sources of parallelism and algorithmic speedup. One of the key bottlenecks is the serialization inherent in forward and backward propagation, which limits strong scaling in the limit. Recently, the parallel-in-time method, multigrid-reduction-in-time (MGRIT), has been applied to some DNNs to overcome this bottleneck by providing new parallelism in the layer dimension (layer-parallelism). This new parallelism is made possible by a connection between the layer-dimension and a hypothetical time-dimension. In this talk, we introduce layer-parallelism with MGRIT and then discuss TorchBraid, which is a high-performance implementation of this approach that supports MPI-based parallelism in combination with GPU acceleration. To achieve this, TorchBraid integrates the PyTorch neural network framework with the XBraid time-parallel library. We present results for Torchbraid with and without GPU acceleration, considering Tiny ImageNet and MNIST, as well as recurrent neural networks and transformers for language processing. We also present new results showing the computational advantage of combining layer-parallelism with data-parallelism and how to adapt standard deep learning techniques, like batch-normalization, to the layer-parallel setting. Lastly, we discuss TorchBraid's approach for overcoming the algorithmic challenges inherent in combining automatic differentiation with layer-parallel in a distributed MPI setting. Overall, TorchBraid enables fast training of DNNs, both in a strong and weak scaling context.

Zoomlink:
https://tuhh.zoom.us/j/81920578609?pwd=TjBmYldRdXVDT1VkamZmc1BOajREZz09

Tail processes and tail measures are important concepts in the theory of regularly varying (heavy tailed) time series. In this talk we will show that these concepts are intimately related to Palm theory of stationary random measures. To motivate the topic, we start with providing some background on regularly varying time series. Then we shall introduce tail fields in an intrinsic way, namely as spectrally decomposable random fields satisfying a certain space shift formula. The index set is allowed to be a general locally compact Hausdorff Abelian group. The field may take its values in an Euclidean space or even in an arbitrary measurable cone, equipped with a pseudo norm. We characterize mass-stationarity of the exceedance random measure in terms of a suitable version of the classical Mecke equation. As a rule, the associated stationary measure is not finite. We shall show that it is homogeneous, that is a tail measure. Finally we will establish a spectral representation of stationary tail measures.

In this talk, we characterize quantitative semi-uniform stability for $C_0$-semigroups
arising from port-Hamiltonian systems, extending and complementing recent results
on exponential and strong stability. Using this characterization, we construct a sim-
ple, universal class of port-Hamiltonian systems that exhibit arbitrary decay rates
slower that $t^{1/2}$. This construction leverages results from the theory of Diophantine
approximation.
This is joint work with Felix Schwenninger (Twente), Ingrid Vukusic (Waterloo),
and Marcus Waurick (Freiberg).

For the prediction and study of water flows in a river or channel the knowledge of the bottom topography - the bathymetry - is required. Direct measurements of bathymetries are possible, but can be very expensive and time consuming. This motivates the development of methods to reconstruct a bathymetry numerically. In my thesis, an observation of the free surface level is used for this reconstruction. By defining an optimisation problem that is constrained by the one-dimensional shallow water equations it is possible to obtain an approximation on the real bathymetry. In this context, the use of Parallel-in-time methods is investigated in order to accelerate the computations.

Zoomlink:
https://tuhh.zoom.us/j/81920578609?pwd=TjBmYldRdXVDT1VkamZmc1BOajREZz09

Transformers have been a major breakthrough in Natural Language Processing (NLP) due to their ability to capture long-range dependencies through self-attention. However, the (self-)attention mechanism suffers from massive memory consumption, especially for tasks with large context windows and high resolution data, such as climate data.

In this talk, I will present a transformer model that uses multiple icosahedral grids to enable large (physical) context windows and high resolutions for various climate-related modelling tasks.

As the model is still under development, the presentation will focus on its technical foundations and properties such as resolution independence and multi-scale output in the context of climate data.

Zoomlink:
https://tuhh.zoom.us/j/81920578609?pwd=TjBmYldRdXVDT1VkamZmc1BOajREZz09

Spectral deferred correction (SDC) is a time-stepping method where fully implicit Runge-Kutta methods (RKM) are solved iteratively. The method is only marginally more complicated to implement than the more ubiquitous diagonally implicit RKM, and it is often simpler for obtaining high-order solutions. We present numerical experiments that show SDC to be a modern and HPC capable method with various advantages over other RKM, including efficient time-parallelisation extensions. To this end, we present adaptive step size selection algorithms for SDC and demonstrate that they boost computational efficiency and resilience against soft faults at the same time. Then, we show that the parallel-in-time algorithm diagonal SDC can be used to extend strong-scaling capabilities beyond the saturation point of space-only scaling. This enables our space-time parallel Python code for the Gray-Scott equation to scale to the entirety of the JUWELS booster machine.

Zoomlink:
https://tuhh.zoom.us/j/81920578609?pwd=TjBmYldRdXVDT1VkamZmc1BOajREZz09

Zoomlink:
https://tuhh.zoom.us/j/88999585223?pwd=YsLVQkzLogvKJt6NNFN2x6OypSvbc9.1