Local Explanations for Deep Learning Models

Six paper discussions, five in-class group activites, exam, and poster presentations must be attended in person. Given that this makes 13 class session in total, the instructor does not recommend enrolling in this class if you can’t attend it in person.

• Train a transformer-based model for an NLP or computer vision application, obtain its predictions, and apply common explainability methods (covered in lectures) to explain the predictions. Four homework assignments are designed to work on this.

• Define utility/function of an explanation for a given application. This course should teach you to ask: “What will best accomplish the explanatory functions [I hope to achieve in this case]?” (specific; application-based) instead of “Is this an appropriate explanation [in this case]?” (generic; very common in current ML research). A successful project proposal (a component of the grade) has to define an appropriate function.

• Identify and apply a method that creates appropriate explanations for a desired utility/function. A successful intermediate project status report should identify such a method.

• Conduct a user study to evaluate produced explanations based on their utility. The course has a dedicated session for preparing an interface for a user study and another session for conducting it with class peers. Having a user study ready ready for the class is a component of the project grade as well as a successful final project presentation that shows how well explanations accomplish the explanatory function according to the study.

• Read cutting-edge research publications requiring being acquainted with the principles and concepts of explainable ML. There will be six class sessions devoted to paper discussions. This part of the course will be organized as a role-playing paper reading seminar with 5 regular roles: original author, scientific peer reviewer, archaeologist, imaginative researcher, and original author of a related paper. Those who are not assigned to be the official presenters will either play a wild card role (any role they’d like to be, such as an industry practitioner, a cranky researcher, etc.) or complete a written artifact (a study notes, a blog post, an opinion piece, a scribe of the class discussion). This format requires everyone to actually read the papers and engage in the discussion about them. The paper discussion is one of the main components of the grade.

• Follow an *ACL/EMNLP (leading NLP conferences) review style to properly review a research paper on explainable ML. One of the roles in the role-playing paper discussion is “scientific peer reviewer”. You will take that role once and your review will contribute to your paper discussion grade.

Your performance in this course will be evaluated by:

• programming homework assignments (30%)
• in-class paper discussions (30%)
• project (30%)
• final exam (10%)

We will almost exclusively talk about applications in NLP (so text) and in computer vision with static images. Inputs in these domains are represented with embeddings—high-dimensional vectors of floating point numbers whose individual dimensions are not interpretable. If you are interested in applications that fall under data science, you will instead likely work with “meaningful” features such as income of a person or zip code of a certain location. This course is not about such applications. We hope you find the course useful even if the data you work with is handled differently from text or images. We believe it can be inspiring to think whether these methods can be applied to a different domain, and realizing that they cannot, can be useful.

We focus on deep learning models (deep neural networks) and we will almost solely talk about transformer-based models.

This course will not focus on global methods that analyze models’ behavior and internals such as probing. We focus on methods that answer questions such as:

• Which part of the input led to assigning this label?
• How to edit the input to change the model’s answer to something else?
• In plain English, why is this input assigned this label?
• Which training examples caused the prediction?

This course doesn’t have formal pre-requisites because these days one can learn about machine learning and adjacent topics in many different ways, but we expected that you…

• …are experienced with programming in Python,
• …are comfortable with basic calculus, probability, and linear algebra,
• …have solid machine learning foundations,
• …have some familiarity with pytorch,
• …are acquainted with Deep Learning 101.

If you completed CS 5353/6353 (Deep Learning) or CS 5340/6340 (Natural Language Processing) or CS 5350/6350 (Machine Learning), we expect you will be able to keep up.

My advice: If you are interested in the course, give it a try. We will spend the first two weeks going over the background and have a graded programming assignment about it. If you struggle with the background concepts and the first homework, you can withdraw—students may drop a course within the first two weeks of a given semester without any penalties.

Revisiting/polishing your knowledge. You can prepare by:

1. There are a ton of Python resources for people with some programming experience. Check them out here. My colleagues suggest these: 1, 2, 3, and 4.

2. Math and machine learning basics are nicely covered in the first part of the Deep Learning book. Obviously, you can use the same book to familiarize yourself with deep learning, especially with the contents of Chapter 6 and Chapter 8 that are a must for this course.

3. Deep Learning with PyTorch: A 60 Minute Blitz (highly recommended)

4. Practical Deep Learning for Coders by Fast.ai (3: Neural net foundations; 5: From-scratch model, 13: Backpropagation & MLP, 14: Backpropagation)

5. Getting started with NLP for absolute beginners is good to familiarize yourself a bit with NLP. During the semester we will use huggingface