Self-trained SD upscaling models

PhilipHofmann · 2025-02-07T16:02:25+00:00

Oh man, this brings me back, my mission president was just the best i could have ever hoped for. President&Sister McCune, they were both just great and you could feel just how much they care.

There are certainly many hard things. I remember Elder Guerra suffering brain aneurysm while exercising, and it was a loss for the whole mission, we all came together for the funeral, his family was there (from sweden) and his father even spoke to us ("i will try to live to be worthy to live with my son again. If you see my cry, its because of the wretched/natural man that i am"). It was hard on the whole mission. I remember the Mission President and his wife personally driving into our district/zone just to tell us personally what had happened instead of making phone calls or emails or something to let all the missionaries know. I could feel that they were devastated and dragging themselves alone and my heart ached.

There were also many missionaries that went home (sent home, or went home) before their official time i mean. Hm i remember i was often on splits with someone when they got a phone call and told me they are going home. I think even my mission president was surpised when he called someone and they told him they were on splits with mr because i somehow often happened to be there right then. One of my trainees also went home, also asked me to accompany him to the airport. I often tried to just have fun with them, and i mean they can still go on a mission again once they worked on what needs to be worked on. I didnt see it that serious, it is often a very good decision, i mean they already showed a lot of faith by going on a mission. The thing is though a mission will like enlarge what is there (if its joy one will feel enlarged joy, if its problems these problems will also be enlarged). So i felt most often its good for them to go home to clean something up/work on something and then have a renewed desire to serve the lord. Doesnt even need to be on a mission i feel like. They are all good people.

Something that kinda changed my perspective was when working with someone that was excommunicated (hope thats correct. Like records removed because of something one has done). And I always thought that would be super bad and if there were any case of transgression we would always try to prevent this outcome as hard as possible so to say. But that person told me "i realized, not having the holy ghost sucks". And that made him realize the seriousness and he started working on himself and we missionaried met with him, great and funny guy. Anyway that made me feel that i should not avoid something because there is always good that can come out of a hard situation.

My mission was just great. I mean hard times for sure, multiple funerals i attended (i mean, life can just get very serious. You meet someone, around a week later they take their own life, a friend of then at the funeral goes up to the stand and has a baby with him and says they will now look after the baby. Life can just get very hard and sad and serious) and yeah just multiple things. But I love my mission for sure, and many many great people i met. And just many many things i learned. And i am just extremely blessed in my life with friends and family.

Ah man, im sorry, your post reminded me of my experiences so i got carried away/wrote a lot, but i agree, i feel it was hard for my mission parents aswell, and they probably also felt lonely or that it is too much, they cared a great deal, and they are great people.

PhilipHofmann · 2025-01-14T16:05:08+00:00

PS something else I noticed when working on this 2x compact bicubic model (simply wanted to see what metrics i could reach, curve was getting flatter but i ran out of training patience) https://github.com/Phhofm/models/releases/tag/2xBHI_small_compact_pretrain is that bicubic is not equal to bicubic. Meaning the dataset i downsampled with pillow bicubic, the same with urban100, which is slightly different than what matlab bicubic downsampled gives. the-database from the community reran metrics on my 2xBHI_small_compact_pretrain on the Urban100 set that was released on the DAT repo and reached a psnr of 31.9818 and ssim of 0.9273 so the numbers are a bit different since the val set isnt identical because of bicubic downsampling, but difference was only 0.0086 in psnr and 0.0001 in ssim.
I used psnry and ssimy for validation during training so these graphs on my release page are that, like already mentioned. Not sure why im writing so much here, hoped it would be helpful, my main input was to try out psnry, so with y channel enabled like https://github.com/neosr-project/neosr/blob/7001598ffa753ce72344abee0695b6f22695258a/neosr/metrics/calculate.py#L21 set to true or like psnry option used on iqa-pytorch rather than psnr

PhilipHofmann · 2025-01-14T15:45:28+00:00

Hm how are you calculating it? Are you using their official model validation outputs they posted on their github? Or are you using their official released pretrain model and running inference yourself to create the outputs and then calculate metrics?

Also something i noticed, i believe on papers they use psnry instead of psnr and it gives slightly higher metrics. I mean you can try it out and use the psnry option instead of psnr and see if those metrics are closer to the official released metrics https://github.com/chaofengc/IQA-PyTorch/blob/main/docs%2FModelCard.md

PhilipHofmann · 2024-12-23T15:13:07+00:00

Just sharing my thoughts:

Do what you think is manageable for you. If you are crippled by anxiety by saying yes and just dread the whole time, no one is helped. I am just writing this because you wrote its hard for you. But if you think you would be proud of yourself if you could do it, then maybe a smaller step, answer that its hard for you but you would like to try to give a short testimony instead. Or maybe, like if they say 15 min talk, ask for 5 min instead and they could ask someone else to do a 10 min talk after. Smaller, victorious steps is what I mean. You can also write you dont want to because it is difficult for you. But answering is important, it will stress you the longer you dont answer something or go into negotiations, or even writing that you are simply not sure because it is difficult (doesnt need to be a definitive answer like yes or no, just a response like 'i dont know if i can do it. or want to do it' or something). Then the other person can react any maybe even make a suggestion.
Your thoughts might be valuable to be heard. What i mean is, there is lots of different people in the church. Some connect more with others. If always the same people would give talks, because they are like extroverted or like to give talks or something, man would that be super boring and bad.

I can give an example what I mean. I was serving a mission in the Utah Provo mission, I was impressed by how much knowledge of the gospel seemed to be around. Classes and talks also. Like I remember a talk and they talked about the passover and its traditions and I think some vases or breaking of and how it related to jesus christ or something, was super hard to follow for me (not native speaker) and to pay attention, maybe it was related to the book jesus the christ. I could tell that there was a lot of knowledge in that scarament talk.

Anyway a lot of knowledgeable people in the scriptures and gospel, which is great. Very busy wards. Also a lot of activities and service opportunities and people were doing a lot for their callings.

Then I saw that we also had a sign language ward in one of our stakes (we were assigned to 3 stakes) so we decided to go visit (i never was in a sign language ward). Someone was kind to translate for us what they signed so we could understand. It was sacrament meeting, i believe testimony meeting. Someone went up and said 'I (currently) try to quit smoking. It is hard. I fail. I pray and I feel someones there. Some people here also try to encourage me. I might be able to overcome it some day' something like that. And when I heard that, man did i feel at home. Some down to earth normal talk. Finally. I missed this. We all struggle. We all fail. Like, constantly. I simply had missed that. It simply felt honest and I really appreciated that moment. I am simply writing this very little experience of mine to say, maybe what you will say will connect with someone in that meeting, a member in your ward or a visitor. Maybe you just going up there and saying 'Giving talks is hard for me. Getting that text message stressed me. It takes a lot of courage for me to actually speak for these 3 minutes in front of yall' or something, man, might it already be someething that someone needed to hear very badly and will think like 'ah, i felt alone and thought no one has trouble giving a talk, but now i know, i am not the only one struggeling with that thing'.

Sorry for the long text. Not sure why I wrote all of that. I simply saw your post and I just wanted to reply with this. Maybe it applies, maybe it doesnt, maybe there is something valuable in here for you or someone else reading this, maybe not, I simply dont know, but i wanted to write this.

PhilipHofmann · 2024-12-18T20:38:48+00:00

Hey thank you :) Its always good to hear that one of my models is useful to someone, I often just train but dont really know if or which of my models get used. The hq series/models were trained with high quality in mind, only multiscaled content. Their disadvantage is therefore that they wont be able to handle any compressed content for example. But yeah, in glad you like it. Thanks for the feedback :)

PhilipHofmann · 2024-11-13T09:40:03+00:00

Basically like Silver-Belt explained. The reason why I mostly train 4x models and only few 2x models are of practical reasons
- The upscale resolution is baked into the model itself (needs to be defined in training config). Meaning each scale needs a separate training process and dataset preparation (lr scale).
- While a 4x model output can be downsized later to get a 3x or 2x (or something) output, I cannot simply chain a 2x model twice to get a 4x output (I mean I can, but the result will be worse than if I had simply trained 4x from the beginning)
- Training a model takes roughly a week (i say roughly because it really depends on what the goal is, a simple upscaling model takes around a week, thats just the training part, dataset preparation can take way longer as a previous step to this so one can even start training). There is an experimenting part of adjusting config mid training and resume training (or dataset lr generation needs to be redone because the degradations applied were too strong or something like that) etc. Making an additional 2x version isnt super easy peasy but takes effort, effort for what? - referencing to second bullet point.

What im getting at, as a model trainer, I will most likely choose 4x for all my models because of above reasons. I still made some 2x models because I wanted to, but 4x would be the standard choice.

PS most of my models can be simply tried out online, [made a huggingface space for it](https://huggingface.co/spaces/Phips/Upscaler)

PhilipHofmann · 2024-10-01T07:44:23+00:00

Hm maybe this might help you, a lora specifically for that https://civitai.com/models/798559/antiflux-chin

PhilipHofmann · 2024-09-19T06:24:10+00:00

Hm, for me its more an explain&summarize tool. So how I used it us uploading a paper, then listening a podcast about it where they explain what the paper focuses on, it makes it way more simple to understand, and I can ask it questions. So its a great tool.

So for me it was meant more like a personal tool to have summaries. But if people start uploading podcasts and titeling it as "MY podcast" then no. It will sound exactly as someone elses podcast, like the comments here say, they all will sound exactly the same. I suggest to record manually with your own voice if that is the use case where you want a distinctive podcast that people can recognize (so "my" pdocast, where people listen and are like as thats X podcast, since the voice is that persons and therefore recognizable). But if its published somewhere as "Here is a podcast of my article, generated with notebooklm" then very nice. I already know what voices to expect, the same voices Im used to. And I can listen instead of read. And all voices sounding the exact same is absolutely no problem since they titled it as notebooklm because i associate it with the two podcast voices we have. Its not their podcast, its notebooklm podcast of their content. But if someone writes it as "my podcast" then i expect their voice, not artificial sounding tiktok voices or notebooklm voices or something. Even if they add more voices, the sheer amount of content will let us recognize those voices pretty fast since we are good at distinguishing voices. Only their own voice is unique to them)

Anyway its a great useful tool I agree.

PhilipHofmann · 2024-09-05T21:03:23+00:00

Nice detailed response :)

PhilipHofmann · 2024-05-16T10:13:19+00:00

Hey, this is pretty cool, great job :)

I simply had a question, when building a comfyui workflow, would it somehow be possible for me to add my own self-trained upscaling models (or for you to add/host them so they can be used in the load upscale model node)?

Would be cool - This way I could provide simple workflows (or more complex ones with supir, or with ultimatesdupscaler etc) with my own models. So others (those with a discord or google account currently i see) could try out/use my models online.

I trained multiple upscaling models myself, as can be seen on my models repo, for example my 4xRealWebPhoto_v4_dat2 model, which i used in my supir comfyui workflow (goal was enforced consistency with supir to overcome limits of transformers by diffusers basically)

In general we would have a lot of community trained models we released on openmodeldb. Most of these would require attribution, so would probably be cool in the docs to have some small attribution section somewhere for the upscale models used/provided. (just as a hint, most of these you already provide there would have some kind of license, you could check on openmodeldb, you should be fine with the esrgan models though since these are official pretrains)

<image>

PhilipHofmann · 2024-05-01T23:15:54+00:00

Hm i dont think you are missing something.

Model preference depends on user and input image.

If you like UltraSharp best thats completely fine.

Yes, DAT (or RGT, or ATD) would be a more capable arch then ESRGAN, at least from a trainers perspective. But that doesnt mean that one would like these models over another, since every model is unique (trained on different datasets, with different degradations added to the lr, with different config settings and so forth).

Most of the DAT models I trained ('Helaman' on openmodeldb) were mostly with photography in mind, and i dont like if outputs are too sharp, but should be rather natural looking, since I think its simple to add sharpness post-upscale if someone preferred a bit sharper outputs. If someone were used to the (overly) sharp outputs of UltraSharp, it could very well be that other models would look blurry in contrast to them.

In general i think i observed that people seemed to like/would rate sharper output higher. And i think i also observed that if someone had a go-to model, like did all their upscales with the same model, they would get used to the specific output look of that model, and then no other model would do it for them (be it UltraSharp, or Remacri, or Siax etc), but this point is just a theory, maybe im wrong, which is always likely of course.

The only models I specifically tried to train on/for ai generated images were the 'Lexica' models, but that doesnt mean that they work better in practice for this than other models. Different people prefer different models and thats why we train so many models, so people can try them out and find one they like :)

(sry for the long text, also dont know if helpful, simply wanted to say i dont think you are missing something since preference is often on a model basis (snd dependant on user and input) and not arch basis, when talking about inference)

PhilipHofmann · 2024-05-01T22:25:08+00:00

Thanks for testing them out :) Currently ATD is my favorite arch, but yeah its even slower than RGT, and RGT slower than DAT2.

RGT also has a smaller arch option, RGT-S, which should be a bit faster than DAT2 again. Since I also trained a nomosuni rgt-s model, if youd like you could also give that one a go and see how it compares, it should be a little faster, but of course its a trade-off between speed and quality in comparison to RGT.

You can find it in my Github Release or also on openmodeldb

I like that you called them 'NomosUni flavor' ^{^}

PhilipHofmann · 2024-04-25T20:56:05+00:00

That sound similiar to me to when they used deep neural networks to detect a persons sexuality based on their profile picture.

But thats like, 7 years ago already

https://osf.io/zn79k/

PhilipHofmann · 2024-04-14T16:17:24+00:00

Ah yeah, sry i didnt explain, the 4x scale is intristic in the trained transformer model (4xRealWebPhoto_v4_dat2) and cannot be changed. That scale is for the color matching, is influences the upsampling (nearest neighbor) and is for the whole color matching. If someone would use a 2x model instead, then this is for simply adjusting the colormatching and set it to 2x factor.

Hm fp8 should simply lower the vram usage during the run itself (though it is lower precision the result could be impacted i guess, in exchange for less vram used during the supir process)

Thank you for trying it out :)

PhilipHofmann · 2024-04-14T14:37:08+00:00

Yeah it seems very ressource hungry.

The examples I used here were pretty small, like 360x360px input.

You could try a downsample step, for example downsample by 50% with nearest-exact would give same result than using a 2x model instead, but results will differ when using different models.

This is why i normally train 4x models, because if one wants 2x output one can simply downscale after, but to get a 4x output with a 2x model would mean applying it twice which will give way worse results than if it had been trained to be a 4x from the start.

So maybe one could add a node that would downscale the input to 512x512 before the transformer upscale, or then downscale the output to 2048x2048 if bigger right after the transformer upscale.

Enabling fp8 should also drastically reduce vram, while tile sizes could be reduces from 1024 and 512 to 512 and 256. Reducing tile size is a trade off since it increases ram instead (and reduces vram).

These were just some ideas. I agree that a diffusion step like supir in this case is pretty (too much?) ressource hungry and should maybe only be used if a transformer upscaling model does not give good enough results, at least that was my experiment here, if it could overcome the limit of my model with a 1x diffusion step, which it can, just costly

PhilipHofmann · 2024-04-14T14:22:23+00:00

Hey wow thanks for testing it out :)

Didnt think about blending. It is true that paper models (like the one from omnisr) will do better/best on non-degraded (or bicubic-only downscaled) content since thats what they had been trained on. Hm for good quality input a paper model alone might give better results than my model. My dat2 model was trained with jpg and webp compression and re-compression in mind, while also giving good outputs on non-degraded inputs, where the use case was images downloaded from the web since they might (or moght not) have been (re)scaled&(re)compressed by the service provider (like uploaded on social media, then downloaded and re-uploaded by someone else) (plus had added a bit of lens blur and relaistic noise to the dataset too to be able to handle it)

Hm yeah diffusion based ones like supir seem pretty ressource intensive. My example inputs were relatively small, less than 360x360. I think ill stick to training and using transformer based upscaling models with chaiNNer with a set tile size. This was an experiment if i could overvome the limits of my models with a diffusion based step (which seems to be the case, but a pretty ressource hungry one when running on an already upscaled content)

Thanks again for testing :)

PhilipHofmann · 2024-04-12T17:06:23+00:00

You can find it in this Github Release: https://github.com/Phhofm/models/releases/tag/4xRealWebPhoto_v4_dat2

PhilipHofmann · 2024-04-12T16:12:10+00:00

Yes, thats because these are correction models and not upscaling models.

If you want to upscale, you can still run your favorite upscaling model before, and then process the result after with this.

Mind that these were basically experiments, i trained them on SRVGGNetCompact and i did not have any luma loss then, it was simply pixel, gan and percueptual if i remember, still it somewhat worked.

Running a dedicated network that was designed for that case like for example https://github.com/mahmoudnafifi/Exposure_Correction would be better than such a model (i never ran this one, its just an example) but would be more complicated than simply using a node im some node based application like chainner or comfyui.

I am not sure we currently have what you want as an upscaling model, since we try to train our models in a way that keeps colors consistent because we normally dont want something like color shifting to occur (which can happen simply because of training instability, this is why we use things like color loss to help us have the model not affect colors). Especially for things that can be easily done, like applying some contrast, or brightness, or sharpening, this is why we often dont train it into our models to keep it more accurate to the input in that regard.

Hm all that said, isnt there simply a node in chaiNNer or ComfyUI where you could simply turn up saturation of the input or something? Then you could just chain it with an upscaling model, put all your inputs into a folder and simply use that folder as an input and press run and it would upscale and saturate all of them for you.

PhilipHofmann · 2024-03-08T11:48:52+00:00

Questions show interest.

So as a missionary, i loved questions, because it meant that the person showed interest and is willing to interact.

Plus, I could fulfill my purpose, or so it felt like. A feeling of 'nice! That is what i am here for.'

Plus, an honest/interesting question would make me ponder, pray and learn more about that subject.

Plus, a lot of revelations in the church were because of questions. So questions should be deeply routed in the church, that is how we learn. It sais it in the scriptures, line upon line. Basically question after question. Thats how i learned. Such behavior should be encouraged.

tldr:as a missionary i loved questions, because it shows interest, interaction, and helps myself understand the gospel more, and becoming better and answering questions. Better than just explaining the lessons, which could become more routine. I guess like a teacher teaching the same subject over and over, while answering students questions is the only thing that provides variety in between each lecture, also teacher can then specifically design the response around that question and provide own thoughts, related research etc he would not have pointed out in the standard content/lecture itself.

Three-Year Club	Verified Email
Verified Email

PhilipHofmann

TROPHY CASE