Perhaps you saw an earlier post I wrote about deep dreaming Prague pictures, and you said to your self, “self, I wish I knew more about the techniques to make those crazy looking pictures.”
Well you are in luck since I’ve now posted the slides where I attempted to explain these two works to our reading group: 1) Google’s DeepDream, and 2) A Neural Algorithm of Artistic Style.
I already tried to give an intuitive explanation of how #1 Google’s DeepDream works here:
So let’s focus on work #2, A Neural Algorithm of Artistic Style. In this work, the authors separate out the style component of an image from the content part of an image.
Let’s reflect on this rather interesting statement for a moment; we can separate an image into style and content. This is somewhat intuitive as you can imagine two artists, each with a particular style, creating the same scene (i.e., the content) that appears quite different given a particular artists style. While the scene might be recognizable, the style of painting may be quite different.
This insight, that an image can be separated into style and content components, is key to this work.
Given this ability to separate an image into style and content components, we can create a new image, that has the content from one image, and the style from another!
Consider that statement again; we can take the content from image C and the style from another image S, to create a third artificial image A that reflects the content of C and the style of S.
The images C, S, and A, are represented using the responses in the layers of a pre-trained convolutional neural network. The artificial image A is then optimized to have similar content as C and style as S. The exact details of how to do this are discussed within the slides (note that while the math is fairly clear, it’s remarkable that style can be transferred as well as it does. In fact if you watch this great video by Andrej Karpathy, he comments that “it’s kind of magic” that this approach works so well, so don’t feel bad if you too find it strange).
If you’re a bit fuzzy on the concepts of pretrained convolutional neural networks, you might be interested in reviewing earlier slides I created here:
ImageNet classification with Deep Convolutional Neural Networks – [paper explained]
Here is a link to the original paper:
Gatys, L. A., Ecker, A. S., & Bethge, M. (2015). A Neural Algorithm of Artistic Style. arXiv Preprint
https://arxiv.org/abs/1508.06576
I also tried out some preexisting Caffe code to do style transfer,
https://github.com/fzliu/style-transfer
and while I didn’t get the greatest results, the image at the very top of this post hopefully gives you an intuitive sense of what the expected inputs and outputs are. You can see much nicer images in the slides or in the original paper.
And finally, here are the slides: