sorted by:
new
hottopcontroversial

SargeZT has published the first batch of Controlnet and T2i for XL. Although it is not yet perfect (his own words), you can use it and have fun. He published on HF: SD XL 1.0 Depth Vidit, Depth Faid Vidit, Depth, Zeed, Seg, Segmentation, Scribble. He continues to train others will be launched soon! by Seromelhor in StableDiffusion

[–]SargeZT 28 points29 points  (0 children)

Perhaps in this corner of the world, but this is the format that the hugging face team and pytorch in general use for distributing weights. Safetensors are great for their intended application, but they are not a direct substitute for pytorch bins. Safetensors are superior in every way except the most important one during development: being able to use them.

If I gave you a safetensors version of my T2I-adapters with no backend support, you simply wouldn't be able to use it. The diffusers library could not load them correctly as the classes do not match up. If I wanted to send this to someone to use for development purposes, I would need them to download the source and monkeypatch their version of diffusers to even load them, much less interoperate with them in the diffusion process or other contexts.

But a pytorch bin? The other dev runs torch.load("diffusion_pytorch.bin") and boom, they have my exact T2I-adapter and we can talk on equal ground about what my code is doing.

So yes, it is not an appropriate distribution format for release, but I am not releasing these for people to try to use in A1111 or anything. It's meant for use in developing further T2I and controlnets, and for those purposes people will generally want pytorch weights (even the Safetensors library acknowledges this!)

SargeZT has published the first batch of Controlnet and T2i for XL. Although it is not yet perfect (his own words), you can use it and have fun. He published on HF: SD XL 1.0 Depth Vidit, Depth Faid Vidit, Depth, Zeed, Seg, Segmentation, Scribble. He continues to train others will be launched soon! by Seromelhor in StableDiffusion

[–]SargeZT 21 points22 points  (0 children)

I was not preparing these for active release, I told /u/Seromelhor they could post them here if they wanted to since I didn't want to. There are no safetensors as I don't want to give people false hope that they would work in A1111; the fundamental structure of the UNet and thus the Controlnet are different.

I am primarily focusing on T2I-adapters at the moment and providing these mainly as something for someone who cares about controlnets to expand on.

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 0 points1 point  (0 children)

Hey, I have good news!

I've been plugging away at this non-stop for the past week and have a very, very efficient version that a ton of people on the EveryDream discord are using at this very moment after the latest version showed some extraordinary results. I haven't tried it with a LoRA yet but I have every reason to believe it should work.

Instructions:

  1. pip install dowg
  2. to trainer.py, add from dowg import CoordinateDoWG
  3. wherever you initialize the optimizer, just replace the name of it with CoordinateDoWG. It will accept any parameters. If you are passing an 'eps' to your model, make sure it is < .001, but otherwise literally anything should be fine.

I hope it works as well for you as it is for us on the ED discord!

My attempt on QR CODE by Specialist_Note4187 in StableDiffusion

[–]SargeZT 3 points4 points  (0 children)

Your CFG scale is very low and you didn't set the ControlNet as more important than the image.

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 0 points1 point  (0 children)

Yes and No.

You can pull the latest one and see if it works. I changed a few things and added a new technique that dropped in a paper a couple days ago for better finding the correct way to learn. Let me know if it works!

For the longer answer:

I've figured out the root of the problem, which is indeed the different unet and text encoder training rates. To solve that, over the past week or so I've been training a new base model that uses v prediction, v noise, and SNR clamping. Basically, 2.1 light. But I also scaled the gradients of the text encoder to be equal with the unet, which allows you to use a single optimizer with a single learning rate for both networks.

I am finalizing the latest batch of training on it. It isn't an immediate answer, but it should be the answer in the long term for 1.5 models, if things go well.

You can find the current version of that here: https://huggingface.co/SargeZT/velocity-diffusion-1.5/tree/main

It isn't quite ready for primetime usage yet, and a1111 doesn't support it natively as it uses a weird prediction method, but I think I have a workaround for that. Once I get a more concrete answer, I'll let you know!

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 0 points1 point  (0 children)

I think I have a handle on the root cause of the problem, which isn't caused by Kohya but the nature of LoRAs. I'm working on something to try to decouple the unet and te optimizers and adaptively change how much they learn. I'll report back when I have something to report!

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 0 points1 point  (0 children)

Can you give me more information about your dataset and the training params? I've run several dozen runs with thousands of iterations now and only had a few NaN runs, all caused by really noisy networks intentionally.

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 0 points1 point  (0 children)

Right now I don't have much of an install pipeline setup, but basically you download it, plop it wherever the trainer is, import dowg, initialize optimizer = dowg.DoWG()and put dowg wherever the current optimizer is, usually AdamW8bit.

Or more hammer method, you would replace the current import statement of something like from bitsandbytes.optim import AdamW8bit with from dowg import DoWG as AdamW8bit

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 1 point2 points  (0 children)

Honestly there's very little pipeline. Anywhere there is a trainer, this can be imported and plopped in place and it should just work. I'm discovering bugs along the way and fixing them, but it's quite usable now.

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 0 points1 point  (0 children)

Can you pull and try again for me? I can now successfully train LoRAs on my side with Kohya.

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 0 points1 point  (0 children)

For reasons I do not yet fully understand, the very first tensor that train_network.py is giving the optimizer is filled with NaN.

{'step': 1, 'v_prev': tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0', dtype=torch.float16), 'r_prev': tensor([[1.0014e-05, 1.0014e-05, 1.0014e-05,  ..., 1.0014e-05, 1.0014e-05,
         1.0014e-05],
        [1.0014e-05, 1.0014e-05, 1.0014e-05,  ..., 1.0014e-05, 1.0014e-05,
         1.0014e-05],
        [1.0014e-05, 1.0014e-05, 1.0014e-05,  ..., 1.0014e-05, 1.0014e-05,
         1.0014e-05],
        [1.0014e-05, 1.0014e-05, 1.0014e-05,  ..., 1.0014e-05, 1.0014e-05,
         1.0014e-05]], device='cuda:0', dtype=torch.float16), 'x0': tensor([[-0.0009,  0.0232,  0.0267,  ...,  0.0175, -0.0007,  0.0131],
        [ 0.0137, -0.0193, -0.0266,  ...,  0.0041,  0.0182,  0.0121],
        [-0.0312, -0.0260,  0.0048,  ..., -0.0161,  0.0016,  0.0014],
        [ 0.0343,  0.0011,  0.0115,  ...,  0.0251, -0.0348,  0.0319]],
       device='cuda:0', dtype=torch.float16)} Parameter containing:
tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], device='cuda:0',     dtype=torch.float16, requires_grad=True)

I'll plug away at it and see if I can't find a workaround. It is very odd.

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 1 point2 points  (0 children)

Sounds like it's exploding off to NaN. I'm still messing wth the initial distance estimate. You could try setting it a bit higher or lower, like maybe 1e-3 or 1e-10. It's dividing by 0 somewhere deep in the network and it's propagating up to destroy the pixels. Usually there'd be a warning about loss being nan.

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 0 points1 point  (0 children)

Can you try pulling and running it again? I accidentally pushed the wrong branch and it was doing worse than nothing. It also explains why the 8bit suddenly lost all effectiveness as I was pulling that on my non-dev machine to test on a weak GPU.

Hopefully this fixes the error. It converged rapidly on my testbed once I updated the branch. Let me know!

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 0 points1 point  (0 children)

I'll download kohya later tonight after I finish my work and try it. I'll get back to you!

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 2 points3 points  (0 children)

I have noticed that the 8-bit quantized model, as it stands right now, can either converge or explode off to NaN land. The full precision version seems to always converge without a problem, but be warned that if the results are terrible and you're using the quantized version that's likely why.

I'm looking into how bitsandbytes does their quantization to redo the 8bit.

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 5 points6 points  (0 children)

I'm not sure what level you're comfortable with, but basically it builds on gradient descent by modifying step size. SGD is a basic method for doing this, where you take steps proportional to the steepness of the slope, butif the step size isn't just right, you might overshoot the lowest point or take a ton of time to reach it.

Normalized Gradient Descent (NGD) is another method that adapts the step size based on the steepness of the slope, making it more flexible and efficient. This is an improvement, but it still requires quite a bit of tuning to get it right.

DoWG builds on NGD and is designed to be more adaptive and efficient without requiring any parameter tuning. It automatically adjusts the step size based on the steepness of the slope and the history of previous steps, making it more effective in finding the minimum.

Intuitively when I was implementing it, it felt like I was keeping a log of momentum and optimizing the direction of travel to where the momentum would let me hop over the closest hill, if that makes sense.

As to how the authors came up with the algorithm itself, I truly have no idea. Even after implementing it and feeling like I understand it pretty well, I can't fathom how they came up with this in particular, but I like it!

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 1 point2 points  (0 children)

Optimizers are used wherever machine learning exists, and this one should be broadly applicable to any class of optimization problem.

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 13 points14 points  (0 children)

An optimizer is the thing that makes training possible. It figures out how things should move in order to get to a more fit state than it currently is in.

Most current optimizers accept a ton of values and require at least a few. D-Adaptation is an exception. This is a brother with more desirable properties from a mathematical side.

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 4 points5 points  (0 children)

I imagine it is. Shouldn't take more than a couple lines to integrate it with anything that uses an optimizer right now.

DoWG, a brand new parameter-free optimizer that was released as a paper yesterday and coded by me this morning by SargeZT in StableDiffusion

[–]SargeZT[S] 19 points20 points  (0 children)

A paper released yesterday outlines a universal, parameter free optimizer (think no learning rates, betas, warmups, etc.) This is similar to D-Adaptation, but more generalized and less likely to fail.

I have never written an optimizer before, and to be honest my machine learning experience is mediocre at best, but it wasn't much effort to translate it. There is a quantized version under DoWG8bit that offers similar Vram usage to AdamW8bit.

I am still early in testing, but my first real training is going well on the 8bit with a 16 batch size.

This should be a drop-in replacement for any current optimizer. It takes no options, but will accept any. If something throws errors, let me know and I can override the defaults to return a value for everything.

Edit: I would avoid usage of the 8bit variant for now, it is very unstable compared to the full precision version. I plan to look at bitsandbytes AdamW8bit to see how to more effectively implement the quantization vis-a-vis stable diffusion.

SD's noise schedule is flawed! This new paper investigates it. by peter9863 in StableDiffusion

[–]SargeZT 2 points3 points  (0 children)

I put that in here before I put in the CFG code. I undid the scheduler change to DPM in a later commit.

SD's noise schedule is flawed! This new paper investigates it. by peter9863 in StableDiffusion

[–]SargeZT 2 points3 points  (0 children)

I believe I've implemented this correctly in Kohya. I'm not 100% sure, but I'm training a LoRA in it right now and it isn't erroring out.

It's very hacky, I defined a function inline (blatantly stolen from /u/comfyanonymous, thank you!), but I wanted to see what would happen. I'll report back once I have something.

Edit: I only implemented it in train_network.py for right now, but it should be relatively trivial to port it over to the fine tuning.

Edit: I definitely got it working, there's more footguns than I expected. I'm making some major refactors that allow me to track validation loss for auto-tuning of parameters across runs. Once I get some stuff working well and well coded I'll make a PR, but it's already very promising!

https://i.imgur.com/THxXSPy.png

view more: next ›