In neural style transfer, why do we focus on a single style image?

Question

In all the descriptions of neural style transfer that I've seen so far, there is a single style image and a single content image, and the task is to produce a new image with the style of one image and the content of the other. For example, we can make the Mona Lisa in the style of van Gogh's Starry Night.

However, this approach seems limiting to me because it is reproducing the style of a single image and not the overall style of the artist. If van Gogh painted a portrait like the Mona Lisa, chances are he would not have used the same colours and motifs as Starry Night. For example, in the Starry Night Mona Lisa linked above, the skin is bluish with swirls and there are a few star-like artefacts. These features are not present in the original Mona Lisa, and they don't appear in van Gogh's portraits like this one either.

Has much work been done in the direction of reproducing an artist's overall style, rather than the style of a single image?

One solution I am imagining is you could have a database of images in the same style, and to stylise a particular content image you could find the style image which is "most similar" to the content image (for some appropriate definition of "similar") and use that as input to the style transfer algorithm. But I haven't tried this and maybe there are better solutions.

Answer 1

So far it is correct. The style transfer technique from the paper "A Neural Algorithm of Artistic Style" only uses a single image. To be exact, the final loss function that the model uses is the following:

$$ L_{total}(\vec{p}, \vec{a}, \vec{x}) = \alpha L_{content}(\vec{p}, \vec{x}) + \beta L_{style}(\vec{a}, \vec{x}) $$

where $\vec{p}$ is the original image, $\vec{a}$ is the image we want to copy the style from, $\vec{x}$ is the initial random image we are going to iterate on to get our final result. Then $L_{content}$ is a function that calculates the error between the original image and the generated image. Similarly, $L_{style}$ is the loss function that calculates the error between the style image and the generated image.

So this specific method takes two images as input. It can generate another image that copies the style from one and the structure from the other to create a final image with both of those properties.

Why do we focus on a single style image?

The focus is on a single image because there was never a need to focus on multiple images. The goal of the algorithm is not to replicate the style of an artist, but the style of a single image.

Other solutions:

If you want to replicate the styles of multiple images, other generative models could work, like pix2pix, VAEs, GANs or diffusion models. These use latent spaces for saving the information from the trained data, to then generate completely new images. The idea would be to use Stable Diffusion-like models.

Your solution could also work - let's use the method from this paper and try a visual encoder to embed each style we want to copy into an embedded space. Then compute similarity with a similarity function to determine which style image best matches the requested original image. In that way, you could be more exact on the type of style that will best match your image.

Tools for creating the systems mentioned

For using generative models, the best current approach is the Stable Diffusion + DreamBooth pipeline, if you want to fine-tune on a specific set of styles given a special token.

In the case of the database solution, using a vector database like Pinecone, that supports image encoding, or Weaviate could work.

It's important to notice that there could be some ambiguity on the meaning of an artist's style. The best we can do is select multiple images, extract patterns from those images, and transfer them into another. If that is what you are referring to, these methods could work, but sadly human cloning has not been invented yet ;) so more than this is going to be quite difficult!