Questionnaire

  1. What is a “feature”?
  2. Write out the convolutional kernel matrix for a top edge detector.
  3. Write out the mathematical operation applied by a 3×3 kernel to a single pixel in an image.
  4. What is the value of a convolutional kernel apply to a 3×3 matrix of zeros?
  5. What is “padding”?
  6. What is “stride”?
  7. Create a nested list comprehension to complete any task that you choose.
  8. What are the shapes of the input and weight parameters to PyTorch’s 2D convolution?
  9. What is a “channel”?
  10. What is the relationship between a convolution and a matrix multiplication?
  11. What is a “convolutional neural network”?
  12. What is the benefit of refactoring parts of your neural network definition?
  13. What is Flatten? Where does it need to be included in the MNIST CNN? Why?
  14. What does “NCHW” mean?
  15. Why does the third layer of the MNIST CNN have 7*7*(1168-16) multiplications?
  16. What is a “receptive field”?
  17. What is the size of the receptive field of an activation after two stride 2 convolutions? Why?
  18. Run conv-example.xlsx yourself and experiment with trace precedents.
  19. Have a look at Jeremy or Sylvain’s list of recent Twitter “like”s, and see if you find any interesting resources or ideas there.
  20. How is a color image represented as a tensor?
  21. How does a convolution work with a color input?
  22. What method can we use to see that data in DataLoaders?
  23. Why do we double the number of filters after each stride-2 conv?
  24. Why do we use a larger kernel in the first conv with MNIST (with simple_cnn)?
  25. What information does ActivationStats save for each layer?
  26. How can we access a learner’s callback after training?
  27. What are the three statistics plotted by plot_layer_stats? What does the x-axis represent?
  28. Why are activations near zero problematic?
  29. What are the upsides and downsides of training with a larger batch size?
  30. Why should we avoid using a high learning rate at the start of training?
  31. What is 1cycle training?
  32. What are the benefits of training with a high learning rate?
  33. Why do we want to use a low learning rate at the end of training?
  34. What is “cyclical momentum”?
  35. What callback tracks hyperparameter values during training (along with other information)?
  36. What does one column of pixels in the color_dim plot represent?
  37. What does “bad training” look like in color_dim? Why?
  38. What trainable parameters does a batch normalization layer contain?
  39. What statistics are used to normalize in batch normalization during training? How about during validation?
  40. Why do models with batch normalization layers generalize better?

Further Research

  1. What features other than edge detectors have been used in computer vision (especially before deep learning became popular)?
  2. There are other normalization layers available in PyTorch. Try them out and see what works best. Learn about why other normalization layers have been developed, and how they differ from batch normalization.
  3. Try moving the activation function after the batch normalization layer in conv. Does it make a difference? See what you can find out about what order is recommended, and why.

In [ ]: