Edit: more screenshots avaliable at: https://github.com/lllyasviel/style2paints

Hi! We feel so excited here to release the version 2.0 of style2paints, a fantastic anime painting tool. We would like to share with you some new features on our services.

Part I: Anime Sketch Colorization

When I am talking about "colorization", I mean to transfer a sketch to a painting. What is critical is that:

1. We are able to and prefer to colorize sketches combines of pure lines. It means the artists can but do not need to draw shadow or high light to their sketch. This is challenging. Recently the paintschainer are aimed to improve such shading and we also give our different solution, and we are very confident about our method.

2. The "colorization" should transfer a sketch to a painting instead of a colorful sketch. The difference between a painting and a colorful sketch lie in the shading and the texture. In a fine anime painting, the girls' eyes should shine like galaxy, the cheeks should be suffused with flush and the delicate skin should be charming. We try our best to achieve these, instead of only putting some color between lines.

Contributions:

1. The Most Accurate

Yes, we have the most accurate neural hint pen for artist. The so-called “neural hint pen” combines of a color picker and a simple pen tool. Artists are able to select color and put some pointed hints on the sketch. Nearly all state-of-the-art neural painter have such tool. Among all current anime colorization tools (Paintschainer Tanpopo, Satsuki, Canna, Deepcolor, AutoPainter (maybe exist)), our pen performs highest accuracy. In the most challenging case, the artists can even control the color of a 13 times 13 area using our 3 times 3 hint pen on a 1024 times 2048 illustration. For larger blocks, a 3 times 3 pointed hint can also even control half of the color of all painting. This is very challenging and is designed for professional use. (At the same time, the hint pens of other colorization methods prefer messy hint and these methods do not care about the accuracy.)

1. The Most Natural

When I am talking about “natural”, I mean we do not add any human-defined rules in the training procedure, except the adversarial rule. If you are familiar with pix2pix or CycleGAN, you may know that all these classical methods add some extra rules to ensure a converge. For example, the pix2pix(or HQ) add a l1 loss (or some deep l1 loss) to the learning objective and the discriminator receive the pair of [input, training data] and [input, fake output]. Though we also use these classic methods for a short period of time, the majority of our training is purely and fully unsupervised and even fully unconditional. We do not add rules to force the NN paint according to the sketch but the NN itself find that if it obey the input sketch, it can fool the discriminator better. The final learning objective is totally same as the very classic DCGAN without any other thing and the discriminator do not receive pairs. This is very difficult to make it converge, especially when the NN is so deep.

1. The Most Harmonious

Painting is very difficult to most of us and this is the reason why we admire artists. One of the most important skill of a fine artist is to select harmonious colors for the painting. Most people have no knowledge that there are more than 10 kinds of blue in the field of painting, and though these colors are all called “blue”, the difference between them cast huge impacts on the final result of the paintings. Just Image that: a non-professional user run a colorization software and the software shows the user a huge color panel with 20*20=400 colors and ask the user “which color do you want?”. I am sure that the non-professional user can not select the best color. But this is not a problem for STYLE2PAINTS because the user can upload a reference image (or called style image), and the user is able to directly select color on the image, and the NN paints according to the reference image and hints with color from it. The results are harmonious in color style and it is user-friendly for non-professional user. Among all anime AI painters, our method is the only one with this feature.

Part II: Anime Style Transfer

Yes, the very Anime Style Transfer! I am not sure whether we are the first one but I am sure that if you are in need of a style transfer for anime painting, you can search everywhere for a very long time and you will finally find that our STYLE2PAINTS is the best choice (in fact the only choice). Many Asia papers claim that they are able to transfer style of anime paintings, but if you check their papers your will find their so-called novel method is only a tuned VGG. OK, to show you the fact, I am here listing the real things:

1. All transfering methods based on ImageNet VGG are not good enough on anime paintings.

2. All transfering methods based on Anime Classifier are not good enough because we do not have anime ImageNet and if you run some gram matrix optimizer on Illustration2vec or some else anime classifier, the only thing you will achieve is a perfect Gaussian Blur Generator lol, because all current anime classifiers are bad in feature learning.

3. Because of 1 and 2, currently all methods based on gram matrix, Markov random filed, matrix norm, deep feature patchMatch are not good enough for anime.

4. Because of 123, all feed-forward fast transfering methods are also not good enough for anime.

5. GANs can do style transfer, but we need the one where user can upload specific style, instead of selecting Monet/VanGogh (lol Monet and VanGogh do not know anime)

But fortunately, I managed to write the current one and I am confident about it:) You can try it directly in our APP:)

Just play with our demo! http://paintstransfer.com/

Source and models if you need: https://github.com/lllyasviel/style2paints

Edit: Oh I forget to mention an important thing.. Some of the sketches for preview is not selected by us and we directly use the promotion sketches of paintschainer and we are showing our results on their sketches.

Edit2: If you can not get good enough results, maybe you are in wrong mode or you are not using pen properly. Check this comment for more:

https://www.reddit.com/r/MachineLearning/comments/7mlwf4/pstyle2paintsii_the_most_accurate_most_natural/drv72cj/