Abstract:We investigate how generated structures of GANs correlate with their activations in hidden layers, with the purpose of better understanding the inner workings of those models and being able to paint structures with unconditionally trained GANs. This gives us more control over the generated images, allowing to generate them from a semantic segmentation map while not requiring such a segmentation in the training data. To this end we introduce the concept of tileable features, allowing us to identify activations that work well for painting.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: how to better understand the internal working mechanisms of these models by visualizing and manipulating the activation vectors in the hidden layers of generative adversarial networks (GANs), and be able to draw images with specific structures using GANs trained unconditionally. Specifically, the researchers hope to achieve the following goals:
1. **Understand the internal working mechanism of GANs**: By analyzing the association between the generation structure and the hidden - layer activation, gain a deeper understanding of how GANs generate images.
2. **Generate images with controllable structures**: Be able to generate images from semantic segmentation maps without relying on the semantic segmentation in the training data, thus having greater control over the generated images.
3. **Introduce the concept of tileable features**: Identify those activation vectors that are suitable for drawing, making the structure of the generated images more visually natural.
To achieve these goals, the researchers proposed two methods to visualize the activation vectors:
- **Set activation vectors globally**: Set the activation values at all spatial positions to the same activation vector \( \mathbf{v} \), which is suitable for some "tileable" activation vectors.
- **Set activation vectors based on grids**: Set different activation values at different positions on the grid, which is suitable for "non - tileable" activation vectors.
In addition, the researchers also explored the method of using activation vectors for painting, that is, generating images with specific structures by replacing the activation values in the hidden layers. This provides a new way to generate labeled data, especially in the semantic segmentation tasks of pathological images.
### Formula Representation
Some of the formulas involved in the description are as follows:
- **Set activation vectors globally**:
\[
\text{Output} = G(\mathbf{X}) \quad \text{where} \quad \mathbf{X}_{i,j} = \mathbf{v} \quad \forall i, j
\]
Here, \( \mathbf{X} \) is the activation matrix of the hidden layer, and \( \mathbf{v} \) is the activation vector to be visualized.
- **Set activation vectors based on grids**:
\[
\mathbf{X}_{i,j} =
\begin{cases}
\mathbf{v} & \text{if } (i,j) \in \text{blue pixels} \\
\mathbf{X}_{i,j}^{\text{original}} & \text{otherwise}
\end{cases}
\]
Here, \( \mathbf{X}_{i,j}^{\text{original}} \) represents the original activation value.
### Conclusion
Through the above methods, the researchers can not only better understand the internal mechanism of GANs, but also generate images with specific structures, which is of great significance for tasks that require a large amount of labeled data (such as pathological image segmentation).