Workshop 1

Contact Arya Mazumdar for further details.

• Title: Challenges in Distributed Learning and Optimization
• Date: April 8, 2021.
• Organizers: Arya Mazumdar (TRIPODS Institute and UCSD), Ananda Theertha Suresh (Google), and Gauri Joshi (CMU)
• Summary: In recent years, the computational paradigm for large-scale machine learning and data analytics has shifted to a massively large distributed system composed of individual computational nodes (e.g., ranges from GPUs to low-end commodity hardware). For example, modern large-scale distributed systems such as Apache Spark and computational primitives such as MapReduce have gained significant traction, enabling engineers to execute production-scale tasks on terabytes of data. A typical distributed optimization problem for training machine learning models has several considerations. The first of these is the convergence rate of the algorithm: after a number of iterations, how close are we to the optimum solution? While the convergence rate is a measure of efficiency in conventional systems, in today's distributed computing framework, the aspects of communication are at odds with the convergence rate. Even though the ML algorithms are highly scalable, with high dimensionality of data and increasing number of servers (such as in a Federated learning setup), communication becomes a bottleneck to the efficiency and speed of the learning. In recent years various quantization and sparsification techniques have been developed to alleviate the problem of communication bottleneck. The goal of this workshop is to bring these ideas together.