| Despite their impact on computer vision and face recognition, the inner workings of convolutional neural networks (convnets) are considered uninterpretable. To counter this, we propose prediction gradients to understand how convnets encode concepts. Existing efforts to understand convnets focus on visualizing units and classes in pixel space. In contrast, prediction gradients measure how individual units in convnets contribute to prediction, and is thus more general than pixelization methods. Calculation is also extremely efficient. Experiments verify the ability of the prediction gradients to measure performance impact of units in a convnet. Using a standard face recognition data set, we also show concepts are distributed in convnets. Finally, we show how the prediction gradient can find the most distinguishing features of faces, what part of one image looks like a target class, and how to use that information to fool classifiers. These use cases demonstrate that prediction gradients can provide unique insight into how neural networks operate. |
