Lachlan MacDonald, “Towards a formal theory of deep optimisation”
“Towards a formal theory of deep optimisation”
Lachlan MacDonald, PhD
Postdoctoral Fellow
Australian Institute for Machine Learning
Abstract: Precise understanding of the training of deep neural networks is largely restricted to architectures such as MLPs and cost functions such as the square cost, which is insufficient to cover many practical settings. In this talk, I will argue for the necessity of a formal theory of deep optimisation. I will describe such a formal framework, introduced recently by myself and collaborators, which elucidates the roles played by skip connections and normalisation layers in global optimisation and facilitates the first proof that a class of deep nets can be trained to global optima with the cross-entropy cost. I will outline how the theory can be applied even to practical architectures such as ResNet in predicting architectural interventions that accelerate training, on practical datasets such as ImageNet. I will conclude with a discussion of intriguing directions for future research stemming from our work.
Biography: Lachlan was awarded his PhD in pure mathematics in December 2019. After 18 months as a postdoc in pure mathematics at the Australian National University and the University of Adelaide, he joined the Australian Institute for Machine Learning in 2021, where he has been researching equivariance, optimisation and generalisation.
Tuesdays, 12pm-1:15pm
Held virtually in person at Clark 110 & over Zoom
Check for event details: https://www.minds.jhu.edu/events/calendar/
Join Zoom Meeting
https://wse.zoom.us/j/98624413365
Meeting ID: 986 2441 3365
One tap mobile
+13017158592,,98624413365# US (Washington DC)
+16469313860,,98624413365# US
Dial by your location
+1 301 715 8592 US (Washington DC)
+1 646 931 3860 US
+1 309 205 3325 US
+1 312 626 6799 US (Chicago)
+1 646 558 8656 US (New York)
+1 669 900 6833 US (San Jose)
+1 719 359 4580 US
+1 253 215 8782 US (Tacoma)
+1 346 248 7799 US (Houston)
+1 386 347 5053 US
+1 564 217 2000 US
+1 669 444 9171 US
Meeting ID: 986 2441 3365
Find your local number: https://wse.zoom.us/u/asoOElnUp
Join by SIP
Join by H.323
162.255.37.11 (US West)
162.255.36.11 (US East)
115.114.131.7 (India Mumbai)
115.114.115.7 (India Hyderabad)
213.19.144.110 (Amsterdam Netherlands)
213.244.140.110 (Germany)
103.122.166.55 (Australia Sydney)
103.122.167.55 (Australia Melbourne)
149.137.40.110 (Singapore)
64.211.144.160 (Brazil)
149.137.68.253 (Mexico)
69.174.57.160 (Canada Toronto)
65.39.152.160 (Canada Vancouver)
207.226.132.110 (Japan Tokyo)
149.137.24.110 (Japan Osaka)
Meeting ID: 986 2441 3365