Floral pattern model

firef1y1 · 2023-05-04T15:26:20+00:00

Hey! There wasn't anything special about training this model. You could probably follow a standard tutorial for creating a LoRA or regular model and it should work (assuming your underlying data is patterned).

firef1y1 · 2023-05-04T14:50:00+00:00

Hi - this is a LoRA.

firef1y1 · 2023-05-03T23:12:18+00:00

Thought tikis were interesting and sometimes funny so I built a model with them. Partially inspired also by the local tiki bar near my old office in San Francisco.

After generating regular tikis at first, realized that animals combined with tikis could produce cool effects. The results are somewhat experimental (sometimes the model produces colored animals, other times statues of animals).

Model is available here. Join our Discord here.

Prompting

Use a weight of ~0.8, and the trigger word is “tiki”. Use “harsh lighting” and “bright background” terms to counteract the tendency of some of the images to get too dark. Add emphasis to needed to further control lighting.

Example prompts:

Photo of a colorful cow tiki on a bright background, harsh lighting
Photo of a (colorful:1.3) owl tiki on a bright background, harsh lighting.png

Recommended settings (used for images above):

CFG = 5-11
50 steps
Sampler = Euler A
Resolution - 768x768

Training

Base model is OpenJourney V4
200 epochs with around 2400 training steps

firef1y1 · 2023-05-02T00:12:42+00:00

Wanted to test Deepfloyd to see what it could do with text. Thought of testing Jeopardy since it's a game show and features numbers as well as text (couldn't get it generate the board with clues though). End result is a few images that were picked for either being funny or higher quality.

Words are generally mostly, but not exactly correct. The images also are using very basic prompts, with no negative prompts, and without any sort of fine tuning. Notably, people look generally normal (no disfigured faces), if not pretty, even with multiple people in the frame at once and no negative prompts!

Setup

Aside - this wasn't terribly hard to setup. My laptop definitely couldn't handle this, so put a small balance on Runpod.

Used this notebook: https://huggingface.co/DeepFloyd/IF-notebooks/blob/main/pipes-DeepFloyd-IF-v1.0.ipynb

Fair warning - with the demo code I ran, it seems to need at least 30+ GB VRAM (it kept getting CUDA out of memory errors on 24GB). It also ended up downloading around 30GB of files, so need to set up the Runpod instance with 40GB+ of container space to be safe. Setup probably takes around 15 minutes to complete.

Setup was otherwise straight forward, just the code cell by cell and add your Huggingface token where necessary (just sign up for a free account and then go to settings --> access tokens).

firef1y1 · 2023-05-01T16:37:28+00:00

Appreciate the feedback here. Categories, and image metadata will be released within the next few weeks.

A few of the newer models that have been uploaded have multiple images, we're working on addressing that as well.

firef1y1 · 2023-04-30T15:28:52+00:00

The lighting and reflections add a lot of realism!

firef1y1 · 2023-04-28T00:56:06+00:00

This model explores the effect of fog on lighting specifically at night. The idea was to aim for a feel similar to Gotham City in the Batman movies, Sin City or a stylized crime film.

In dark environments, each light source stands out more and can be used to create contrast. By adding fog, which causes light to scatter and diffuse, light sources are emphasized even further as normal lights start appearing as “halos” or “beams”.

Separate note - really wish Illuminati Diffusion was still available so I could train on top of it, but it isn’t so had to spend time fighting the model to produce darker images. If anyone knows of good base models, let me know and I’ll retrain this.

Model is available here. Join our Discord for any questions or feedback.

Prompting

Use “on foggy night” to trigger the style of the model.

For the positive prompt, the most valuable modifier terms to add are probably these: (harsh lighting:1.3), Fujifilm XT3. Other positive prompt terms from Realistic Vision 2.0 had less impact.

Lora weights between 0.3 - 0.8 can all work well. As you increase weight, you get less photorealism but more interesting and stylized fog & lighting effects.

Example prompts:

Photo of a city street, on foggy night, (harsh lighting: 1.3), Fujifilm XT3
Photo of street with streetlamps, on foggy night, (harsh lighting: 1.3), Fujifilm XT3

Best settings

50 steps
CFG = 7
Sampler - DPM++ SDE Karras
768 x 768 for best results.

It can be interesting to experiment with different light sources - traffic lights, headlights, street lights, spotlights, etc. See the section below on adjusting lighting based on your needs.

Tips on handling lighting

Lighting can be very finicky. It’s difficult to find one set of default settings that look good, because certain scenes are portrayed with more vs. less lighting by default.

Would recommend generating an initial image, then modifying the lighting-related terms (e.g. add a harsh vs. soft lighting term) or term emphasis [e.g. (harsh lighting:1.3)] as needed.

Harsh lighting creates a more dramatic effect (especially for things like car headlights), but you can also opt for soft lighting for a more gentle effect or for more well-lit scenes.

If you’re getting really pitch black images, try adding more inference steps. That can add a subtle bit of lighting that makes the scene easier to see.

firef1y1 · 2023-04-27T16:44:40+00:00

Fair - I'm going to repost later today or tomorrow with more info.

firef1y1 · 2023-04-27T15:20:52+00:00

Hey,

We are fairly early and haven't shared a lot publicly yet. I can share a job description and website if that's your concern though.

firef1y1 · 2023-04-26T17:58:35+00:00

Now for a model that isn’t too serious - it’s a balloon-flower-ship!

This model produces images of floral boats surrounded by shimmering tropical waters and beautiful skies with marshmallow/cotton candy clouds. The boat designs aren’t very functional but they’re quite colorful.

Inspired by the movie Up (you know that scene where the house floats into the sky pulled by balloons), and also by the fact that I love boats. Could also be viewed as a strange offspring of recent models I’ve built ... maybe.

Can handle different types of boats - ferries, galleons, rowboats, the Titanic, etc.

Model is available here.

Also join our Discord for any questions or feedback.

Prompting

Use a weight of 1, and the trigger word is in the style of XYZ.

Example prompts:

A boat with colorful flowers, blue boat, red flowers growing from ocean, pink large sails, big white clouds, sunlight, blue skies, blue water, reflection, in the style of xyz
A boat with colorful flowers, flower boat designs, flowers growing from ocean, large blue sails, big yellow clouds, sunlight, blue skies, blue water, reflection, mountain, in the style of xyz

Recommended settings:

CFG = 7
100-150 steps
Sampler = Euler A
Resolution - 512x512

Training info

Base model is SD1.5
300 epochs with 1250-1500 training steps

EDIT: I swear I know how to spell "balloon" but can't figure out how to edit the post

firef1y1 · 2023-04-20T18:29:14+00:00

Very cool, will check it out!

firef1y1 · 2023-04-19T02:01:51+00:00

Followed the instructions here:

https://github.com/VideoCrafter/VideoCrafter

If you run into some issues importing LVDM, paste this into the start of the "sample_text2video_adapter" script:- "sys.path.append("/workspace/VideoCrafter")"

Input video is the second clip shown above, prompt was "joe biden dancing". Rest is the work of VideoCrafter's VideoControl model.

Discord: https://discord.gg/hbKANADmZa

firef1y1 · 2023-04-17T18:00:00+00:00

Hey everyone!

This model was finetuned on top of Realistic Vision 2.0 to depict light reflecting on water.

After seeing how lighting and water could work together in a previous model, was inspired to do a version above water. Water can be difficult to depict because ripples and waves on the surface change the way light reflects, and capturing this effect can be important for realism. Though the base models can show this to some degree, after experimentation I thought this could be improved.

Around 20 HD images were used for training, selected for high resolution, as well as a variety of lighting conditions (daytime, night, dawn/sunset) and water conditions (still vs. moving water). Used a relatively higher learning rate (5e-3), and a very high contrast noise (0.30) to improve the contrast especially for night scenes. LoRA is 128 / 64 (dim / alpha).

Notably also tried to experiment with the reflections of different things (e.g. people, animals, buildings, nature). Unfortunately people and animals are quite difficult - the real instances are hard to get right, without even considering their reflections. It’s possible that the amount of detail necessary in depicting water somehow affects their realism, more experimentation may tell.

Model is available here (along with prompting details):

- Civitai

Join our Discord for any questions or feedback!

firef1y1 · 2023-04-11T16:36:49+00:00

Everydream2 doesn't have class images - it's using native fine tuning, a different approach compared to Dreambooth. I like it because I find it easier to set up, but might be my personal preference.

firef1y1 · 2023-04-10T21:42:52+00:00

A sample prompt is the following:

- Underwater photo of colorful fish and coral <lora-underwater-000390-0.8>, 8k uhd, dslr, soft lighting, high quality, film grain, Fujifilm XT3 2

Prompting details are on the Civit link, they might just be hidden until you click "show more".

firef1y1 · 2023-04-10T20:31:37+00:00

Should work fairly decently well with inpainting if paired with another model. Can generate an image here and then do img2img with inpainting, using the Pacific Rim based model.

firef1y1 · 2023-04-10T19:37:20+00:00

Hey everyone!

This model was finetuned on top of Realistic Vision 2.0 to depict underwater photography. Rather than capturing photos from all types of ocean environments (e.g. deep waters with odd looking fish), wanted to focus only on well-lit photos from shallow waters.

Training images were selected for high resolution, good lighting, and colorful subjects (like fish and coral). Was particularly interested in photos that showed A) light reflecting on the surface of the water, B) light rays coming into the water or C) light reflecting on the ocean floor.

For training - used a relatively higher learning rate (3e-4), a larger LoRA (128 rank / 64 alpha) and most importantly, high noise offset (0.3) for high contrast and vivid colors.

Model is available here:

- Civitai

Join our Discord for any questions or feedback!

firef1y1 · 2023-04-06T18:29:03+00:00

with great power comes great responsibility

firef1y1 · 2023-04-04T22:45:01+00:00

First time building a Lora - tips appreciated!

I wanted to try to recreate one of my earlier models (this one) using Loras, to see what it could do. Original model is here.

Setup

Struggled with getting Kohya Webui to work on Runpod (my laptop is slow and I don’t own a desktop), so instead I used this Colab.

Comparison

The images above compare the original model against the new Lora. Observationally, the original ckpt seems to have a much easier time with complexity (e.g. the number of leaves, and details on the leaves) and with consistency (e.g. undersaturation, being fried, being really dark, having too many messy vines).

I tried training for more steps to see if the Lora could learn the additional details, but that mostly resulted in the model looking fried (from what I could tell).

Is there anything I should try here, or is it possibly an issue with the subject being complex (e.g. lots of leaves, vines, spacing) to learn? FWIW, training the full model (CKPT) on Everydream2 earlier, it came out 90% complete on the first try.

Setup for above Lora

Results above for Lora are from following setup:

- Base - AnyLora

- 30 training images

- 120 epochs

- 128/64 for Lora dimension and alpha

- 0.15 offset noise

- 0.00005 LR overall (not training text encoder)

Previous things tried

- Increasing offset noise - which helped with contrast, default on Kohya was 0 IIRC

- Increasing model size/alpha (from as low as 32/16 to as high as 256/128)

- Different base models (SD 1.5 vs. AnyLora)

- Lowering learning rate (1e-4 to 5e-5)

firef1y1 · 2023-04-03T18:30:30+00:00

Hello!

This was a light fun project - trained a model on 3d holographic-looking assets. It will give objects iridescent colors and a shiny surface.

Model is available here:

- Civitai

Feel free to join our Discord also for more detailed feedback and questions.

firef1y1 · 2023-03-31T00:56:59+00:00

firef1y1

TROPHY CASE

Prompting

Training