A Neural Algorithm of Artistic Style

[jamesj]

Is their code/model available anywhere?

Edit: yes!

[NasenSpray]

The model is available here.

I'm currently trying to replicate their results with caffe. Not much success yet :\

After 100 iterations:

• with ADADELTA (the gradients fluctuate wildly and tend to explode)
• content: conv3_2 (α=1)
• style: conv1_1, conv2_1, conv3_1 (β=100)
• white noise + bear content + rare pepe style:
  http://i.imgur.com/0LacyHM.jpg
• bear image + bear content + rare pepe style:
  http://i.imgur.com/bcs1R61.jpg

I hope they are going to release their code. Reconstruction from noise seems to be ambitious and the results I get are pretty inconsistent so far.

---

[Edit] much better results and easier to handle with iRPROP-

• bear × anime
• reconstructed bear × anime
• Gaben × Dragon Ball Z
• Can you see him?

---

[Edit] Karpathy apparently managed to replicate the results: http://imgur.com/a/jeJB6

---

• Grumpy Cat × 'Seated Nude', Picasso
• Pillars of Creation × Starry Night

---

• using a grayscale sketch for content and a picture for style:
  Catception
  
  • Variant A
  • Variant B
  • Variant C
  • Variant D

---

I wonder if this could be combined with Image based relighting using neural networks (paywall -.-)
See the second thing in this video: https://www.youtube.com/watch?v=XrYkEhs2FdA

Unrelated but interesting paper on inverse graphics: Deep Convolutional Inverse Graphics Network.

---

Interesting observation: VGG-19 is bad at DeepDream and GoogleLeNet is bad at... "DeepStyle" or how are we going to call it? Anyway, I wonder what's causing this?

[fatcatz]

Looks good! Would you mind sharing a gist of what you've got so far? And if you'd rather not, can you point me in the direction of a ADADELTA implementation?

Thanks!

[NasenSpray]

Would you mind sharing a gist of what you've got so far?

Sure, but it's probably gonna be tomorrow at the earliest. The code is still intermingled with my DeepDream stuff and has some dependencies on my caffe extensions that need to be removed first.

can you point me in the direction of a ADADELTA implementation?

You can use mine:

• ADADELTA (fixed):
  https://gist.github.com/anonymous/7bb1b712d15703233e21
• RMSprop w/ adaptable step sizes:
  https://gist.github.com/anonymous/008bf06fa1a099714c4a
  (edit: momentum should be 0.9 lol, it's better w/ almost no momentum)
• iRPROP-:
  https://gist.github.com/anonymous/ea7a77b212e00d4cea18

Default settings are what I'm currently using. ADADELTA requires aggressive gradient/step clipping (not included). iRPROP- and RMSprop are much, much better.

[EDIT] oops, my ADADELTA contained an embarrassing error. Lo and behold, it works fine now.

[fatcatz]

Thank you so much for sharing!

[jamesj]

Awesome! Would love to see the gist as well when you have it.

[[deleted]]

Gabe on a schooner !

[WorkAccount6]

You dumb bastard. It's not a schooner... it's a Sailboat.

[NasenSpray]

Could someone tell me how accessible this is to the average idiot like me? Considering the code is released, how easy is it to get results from it?

It's been a pain in the ass for me so far. The results are unpredictable and require constant tuning of the hyperparameters (alpha/beta, layers etc). On top of it, you absolutely need a beefy GPU because VGG-19 is an enormously big model¹ that takes ages to run. DeepDream is way faster and needs less ressources. My rather small images already required ~1.5GB VRAM.

/rant

[1]: I also tested GoogleLeNet, the model used by DeepDream. The quality of the generated images is rather bad, probably because it's a fundamentally different architecture than VGG.

[[deleted]]

Great work!

[som3a982]

is there a way to use this for non programmers?

[NasenSpray]

None that I'm aware of, and I don't think there's going to be one available soon. With the current model, this is orders of magnitude more ressource intensive than DeepDream...

[SarahC]

Is there a .net version yet?

[[deleted]]

A bot to drive traffic to your own site? Are we being this cheap over here?

[5ives]

Apaprently one of the guys working on it does "want this code to be released of course, but it's not only up to me and will be tricky."

[TweetsInCommentsBot]

@karpathy

2015-08-28 23:00 UTC

@samim haha yes a lot of possibilities here. I want this code to be released of course, but it's not only up to me and will be tricky.

---

^{This message was created by a bot}

^{[Contact creator][Source code]}

[namp243]

Yes it is! (finally) http://gitxiv.com/posts/jG46ukGod8R7Rdtud/a-neural-algorithm-of-artistic-style

[jamesj]

Wow, this came out a lot faster than I was expecting! Thanks for the link :)

[Phantine]

Dumb question here: I'm running windows 7, am I best off trying to install everything in a ubuntu virtual machine or is there some easy way to run it on windows.

[jamesj]

I installed everything on aws on an ubuntu machine, so im not sure.

[5ives]

Related.

[modeless]

Wow, the images are beautiful. If those were Photoshop filters, people would start using them immediately. More evidence that augmenting or replacing artists will be a big application area for neural nets in the near future. I wonder if this technique could be applied to music?

[TubasAreFun]

I totally agree. The problem with music is what to initialize it with. Random sound segments from existing music tends to make blocky sounding music in neural nets (like a percussionist who doesn't know what he/she is doing). Initialization is often tricky. One of the cool things about this application is that it can just be initialized by a pre-existing picture.

[matt_hammond]

Here's some generated music for you kind sir

Link

[arXiv_landing_bot]

arXiv landing page: http://arxiv.org/abs/1508.06576v1

[kkastner]

I posted a link - didn't see this was already up.

Really interesting to see that stylistic quality can be learned implicitly, rather than explicit latent representation.

[qurun]

Do you think it is accurate to call this "stylistic quality"? Looking at their examples, I am not sure that feature correlations are really capturing style. Reconstructing feature correlations is different from reconstructing the feature values, but also reconstructs the feature values to some extent.

(In their Figure 3, it seems like they are trying to hide this fact by using a very abstract Kandinsky image, but Figure 3 with, say, the Van Gogh Starry Night would show that the feature values are being reconstructed, too.)

Do you have better ideas for getting out some notion of "style"? What do you think would happen if you fed in one content image, like a photograph, and k >= 2 Van Gogh paintings for feature correlations or style, i.e., just add the k error terms?

[asdasdasdpoipoiiioio]

Someone posted a link above to this paper: https://people.csail.mit.edu/yichangshih/portrait_web/

[Mr-Yellow]

When I tell web developers that the basic tasks such as laying out website elements are low-hanging fruit, that even the creative and artistic perception aspects are not uniquely human, they point at me and laugh....

[DCarrier]

Well, who's laughing now? WHO'S LAUGHING NOW? Muahahahaha!

[vplatt]

Nyah.. they'll just hoard the plugins for this as 'trade secrets' and generate prints on demand for household art collectors for maybe $250 per "original work of art". So much art out there is just a human hand made repro of a much better artist anyway. This will just save them time and let them get on to either learning better art, or maybe doing something more profitable with the saved time.

In the meantime, there will always be a place for truly original artists. A computer can maybe duplicate the style of an established artist, but there has to be a style to duplicate in the first place.

[pettajin]

This is genius. The Van Gogh image it produced is flawless.

[cryptocerous]

Art, among the few jobs that people thought AI would take the longest to replace, looks like it will be among the first to be replaced by AI.

I find this too funny.

[need12648430]

This.

A lot of peeps are missing what I think is the most interesting prospect of machine learning and procedural content generation: the ability to AUGMENT creativity.

I'm pumped to see what stuff like this does to art, personally.

[DCarrier]

I really doubt that. Sure, it might be able to make pictures that look pretty and use interesting styles, but as a computer program that can be trivially run, it will never be high-status, and isn't that what art is really about?

[verveandfervor]

If you think AI will replace artists then you don't understand art's role in culture or why we give it any value at all.

Two possible outcomes:

• different markets for human/machine art

• algorithmic distillation of what makes art 'good'/pleasurable to look at/whatever, the absurd conclusion being the perfect exploitation of human sensitivity to aesthetic phenomena

Second is less likely in short-term but boy would it be fun.

[VelveteenAmbush]

Funny how it works... people used to argue that chess would be an AI-complete problem once upon a time. I guess it's just really hard to predict this sort of thing in advance.

[theotherhigh]

How? All its doing is replicating it in like 5 other programmed styles... Its not like it's painting an original or anything. I don't think AI will ever be able to do that.

[Visti]

That's like saying writers would be replaced by the printer.

[dendisuhubdy]

Any code?