---

CEng 783 - Deep Learning - Fall 2020

Image of the “METU Tree of Science” generated using the Deep Dream Generator.

Instructor: Asst. Prof. Dr. Emre Akbas; e-mail: emre at ceng dot metu.edu.tr; Office: B-208; Office hours by apppointment
Online communication: (forum, homework submissions) https://odtuclass.metu.edu.tr/
Syllabus: pdf

Announcements

• Dec 13, 2020 - Paper presentation schedule is up.
• Nov 10, 2020 - Initial list of candidate papers for in-class presentations is up. The list will be occasionally updated. Papers will be assigned in a first-come-first-serve basis. E-mail me with your preferences.
• Oct 27, 2020 - Zoom link below will be used for the rest of the weeks.
• Oct 19, 2020 - Zoom link for the second meeting.
• Oct 13, 2020 - Student with these IDs can add the class (in addition to those who are already registered). Please let me know if you plan to drop the class.
• Oct 12, 2020 - Link for the first meeting tomorrow at 9:40: https://metu.webex.com/metu/j.php?MTID=mb3572e876f1f5103303ee52a08b890c4.
• Oct 8, 2020 - Please see my answers to frequently asked questions.
• Sep 29, 2020 - I am receiving many e-mails about taking this course as a special student, as an undergrad, from other departments/universities, etc. I am not able to respond each of them. If you want to take the course, please make sure you come to the first lecture. Lecture hours are not scheduled yet.

Late submission policy

Any work, e.g. an assignment solution, that is submitted late past its deadline will receive -10 points per day delayed. For example, if the deadline is Oct 14, 23:55 and a student submits his/her work anytime on Oct 15, that work will be evaluated over 90 instead 100.

Detailed syllabus

An important note: Lecture slides provided below are by no means a “complete” resource for studying. I frequently use the board in class to supplement the material in the slides.

NOTE: Below, links to slides and homework material are broken because I removed the files. Recent versions of these files can be found in the Fall 2021 version of the course.

Week 1

• Introduction and course logistics.
• Reading assignment for next week: “Chapter 5: Machine Learning Basics” from the book “Deep Learning.”
• Assignment 1 is due Oct 22.

Week 2

• Lecture topics: A high-level introduction to Deep Learning (slides)
• Lecture topics: Machine learning background and basics (1 of 2) (slides)
• The link for the recorded lecture is available on ODTUClass.

Week 3

• Lecture topics: Machine learning background and basics (2 of 2) (slides)
• Colab notebook for the in-class hands-on demo on binary classification, gradient descent, hinge loss.
• Lecture topics: Biological neuron, artificial neuron, Perceptron (slides)
• Adding regularization to the hinge loss classifier: Colab notebook
• Reading assignment for next week: “Chapter 8: Optimization for training deep models” from the book “Deep Learning.”
• The link for the recorded lecture is available on ODTUClass.

Week 4

• Lecture topics: Perceptron, Multilayer Perceptrons, Backpropagation, Activation Functions, Stochastic Gradient Descent, Momentum (slides)
• Colab notebook on building, training and evaluating a basic MLP.
• Reading assignment for next week: LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444.
• The link for the recorded lecture is available on ODTUClass.

Week 5

• Lecture topics: Convolutional neural networks, convolution, connectivity types, pooling, AlexNet, data augmentation, dropout. (slides)
• Building, training and evaluating a basic CNN: Colab notebook
• Initial list of candidate papers for in-class presentations is up. The list will be occasionally updated. Papers will be assigned in a first-come-first-serve basis. E-mail me with your preferences.
• The link for the recorded lecture is available on ODTUClass.

Week 6

• Lecture topics: Convolutional neural networks continued, multiclass hinge loss, derivation of cross-entropy loss, notes on initializing neural networks, implementing backpropagation in a modular way, variants of stochastic gradient methods, adaptive learning rate methods. (slides)
• Reading assignment for next week: He, K., Zhang, X., Ren, S. and Sun, J. Deep residual learning for image recognition. In CVPR 2016.
• The link for the recorded lecture is available on ODTUClass.

Week 7

• Lecture topics: some applications of ConvNets. (slides)
• Building a basic object detector: Colab notebook
• Reading assignment for next week: A. Karpathy’s blog post on recurrent neural networks
• The link for the recorded lecture is available on ODTUClass.

Week 8

• Lecture topics: Recurrent Neural Networks, LSTM, GRU. Some applications of RNNs. (slides)
• The link for the recorded lecture is available on ODTUClass.

Week 9

• A RNN example: solving simple arithmetic operations using a sequence-to-sequence encoder-decoder model: Colab notebook (in Keras)
• Lecture topics: Deep generative modeling (slides)
• The link for the recorded lecture is available on ODTUClass.

Week 10

• Lecture topics: A brief intro to deep reinforcement learning (slides)
• Lecture topics: self-attention, transformers, non-local neural networks, graph neural networks (slides)
• The link for the recorded lecture is available on ODTUClass.

Week 11

• Lecture topics: brief overview on latest trends, limitations in deep learning. (1) Self-attention / Transformers, (2) Self-supervised / Unsupervised learning, (3) - Out-of-distribution robustness, (4) Non-differentiable operations & blackbox optimization, (5) Human-level AI, (6) Better efficiency, (7) Implicit deep learning.
• The link for the recorded lecture is available on ODTUClass.
• Take home exam given.

Week 12

• Paper presentations:
  • “EfficientNet: Rethinking model scaling for convolutional neural networks”, Tan and Le, ICML 2019 presented by Sinan Gencoglu.
  • “Don’t decay the learning rate, increase the batch size”, Smith et al., ICLR 2018 presented by Enes Recep Cinar.
  • “On exact computation with an infinitely wide neural net”, Arora et al., NeurIPS 2019 presented by Yavuz Durmazkeser.
  • “Reconciling modern machine-learning practice and the classical bias–variance trade-off”, Belkin et al., PNAS 2019 presented by Munteha Nur Bedir.
  • “No more pesky learning rates”, Schaul et al., ICML 2013 presented by Arash Sadeghi Amjadi.

Week 13

• Paper presentations:
  • “Dynamic filter networks”, Jia et al., NeurIPS 2016 presented by Berke Tezergil.
  • “CondConv: Conditionally parameterized convolutions for efficient inference”, Yang et al., NeurIPS 2019 presented by Rza Hasanli.
  • “Attention is all you need”, Vaswani et al., NeurIPS 2017 presented by Ahmet Sencan.
  • “BERT: Pre-training of deep bidirectional transformers for language understanding”, Devlin et al., 2018 presented by Firat Cekinel.
  • “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”, Dosovitskiy et al., 2020 presented by Irmak Akkuzuluoglu.
  • “End-to-End Object Detection with Transformers”, Carion et al., ECCV 2020 presented by Duygu Arslan.
  • “Show, attend and tell: Neural image caption generation with visual attention”, Xu et al., ICML 2015 presented by Cahit Yildirim.
  • “Continuous control with deep reinforcement learning”, Lillicrap et al., ICLR 2016 presented by Burak Han Demirbilek.
  • "Dueling network architectures for deep reinforcement learning, Wang et al., ICML 2016 presented by Umut Can Gulmez.
  • “Human-level control through deep reinforcement learning”, Mnih et al., Nature 2015 presented by Melis Ilayda Bal.
  • “Improved techniques for training score-based generative models”, Song and Ermon, NeurIPS 2020 presented by Ibrahim Koc.
  • “Accelerating deep learning by focusing on the biggest losers”, Jiang et al., 2019 presented by Tugba Tumer.
  • "Green AI’, Schwartz et al., Communications of the ACM 2019 presented by Ali Ozkan.
  • "Distilling the knowledge in a neural network’, Hinton et al., NeurIPS workshop 2014 presented by Ali Ozkan.
  • “Small data, big decisions: Model selection in the small-data regime”, Bornschein et al., ICML 2020 presented Nika Rasoolzadeh.

Week 14

• Paper presentations:
  • “Tracking Objects as Points”, Zhou et al., ECCV 2020 presented by Raheem Hashmani.
  • “Semi-supervised classification with graph convolutional networks”, Kipf and Welling, ICLR 2017 presented by Semih Kaya.
  • “Adversarial inverse graphics networks: Learning 2d-to-3d lifting and image-to-image translation from unpaired supervision”, Tung et al., ICCV 2017 presented by Ozhan Suat.
  • “Self-supervised learning through the eyes of a child”, Orhan et al., NeurIPS 2020 presented by Arkin Yilmaz.
  • “A simple framework for contrastive learning of visual representations”, Chen et al., ICML 2020 presented by Sarp Tugberk Topallar.
  • “SoftSort: A Continuous Relaxation for the argsort Operator”, Prillo and Eisenschols, ICML 2020 presented by Dogukan Cavdaroglu.
  • “Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval”, Brown et al., ECCV 2020 presented by Feyza Yavuz.
  • “Towards an integration of deep learning and neuroscience”, Marblestone et al., Frontiers in Computational Neuroscience 2016 presented by Ali Dogan.
  • “PoseNet3D: Learning Temporally Consistent 3D Human Pose via Knowledge Distillation”, Shashank et al., 3DV 2020 presented by Bedirhan Uguz.
  • “Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves”, Metz et al., 2020 presented by Bugra Kaan Demirdover.
  • “A generative adversarial approach for zero-shot learning from noisy texts”, Zhu et al., CVPR 2018 presented by Cemal Erat.
  • “The lottery ticket hypothesis: Finding sparse, trainable neural networks”, Frankle and Carbin, ICLR 2018 presented by Cagdas Cayli.
  • “Decoupled weight decay regularization”, Loshchilov and Hutter, ICLR 2017 presented by Fatih Acun.
  • “GhostNet: More features from cheap operations”, Han et al., CVPR 2020 presented by Ali Ilker Sigirci.

---

Frequently Asked Questions about taking the course

I am receiving too many e-mails about taking the course. Unfortunately, I cannot reply them one by one. Below are answers to some common questions.

Q1: Can I take the course?

A1: There is a huge demand for the course from all kinds of backgrounds. Thanks. However, I have the responsibility to evaluate your learning outcomes and grade you. Therefore, I need to limit the number of seats. Based on my previous years’ experience, this limit will be around 40-45.

Since this is a graduate METU CENG course, I need to give priority for the graduate students in our department. In fact, here is the priority order that I will use to accept students to the class. From high to low priority:

• Grad students from METU CENG,
• A limited number of 4th year undergraduate students from METU CENG,
• Grad students from other METU departments,
• Special students (see http://oidb.metu.edu.tr/ozel-ogrenci, you need to be a grad student in some other university to be eligible).

I must note that the first two categories (METU CENG students) almost fill up the whole capacity. So, unfortunately, there will not be much room for the remaining two categories.

Also, precedence will be given to students who are actively doing research in machine learning and related areas. This course is not a PyTorch or Keras tutorial, we intend to go beyond the “user” level.

You might want to check out the other two DL courses given at Multimedia Informatics and Electrical Engineering departments.

Machine learning background is required. If you have not taken a machine learning course before, please do not take this course.

Fluency in Python is required.

Q2: How can I register for the course?

A2: Come to the first class. I will publicly announce the lecture link on this page. In the first class, I will collect information from the participants and then, will decide (based on my answer A1 above) on who will be able to register. This enrollment list will be announced in a couple of hours following the first class. Students listed in this enrollment list will be able to add the course during the add-drop period.

Q3: I was able to take the course during the regular interactive registration. Should I worry about not being accepted?

A3: No worries. You will stay.

Q4: Can I take this course as a special student?

A4: Possible but unlikely. Please see my answer A1 above.

Q5: The deadline for submitting special student papers is Oct 12 but your first class in on Oct 13.

A5: I know. In previous years, the deadline for submitting special student papers was the last day of the first week but this year it the first day of first week. Unfortunately, this is out of my control.

Q6: Even if I don’t officially register for the class, can I audit it?

A6: I always welcome students to just come and listen. I am thinking on how to best implement this.

---

Last updated on 2021-10-12. Created using pandoc.