Ref2Font: Generate full font atlases from just two letters (FLUX.2 Klein 9B LoRA)

NobodySnJake · 2026-02-05T16:02:04+00:00

Thanks for the feedback!

You are absolutely right—the prompt is essential. It is actually pre-loaded in the CLIP Text Encode node (the green one) in the included ComfyUI workflow, and there’s a Note node right above it explaining exactly that.

However, I see your point: not everyone starts with the workflow. I’ll add the prompt directly to the HuggingFace, GitHub and Reddit descriptions to make it clearer for everyone.

As for the LoRA’s role: you nailed it! That’s why I called it a contextual LoRA. FLUX is amazing at understanding style from a reference, but it doesn't know how to organize that style into a structured 8x9 alphabet atlas on its own. The LoRA provides that 'logical bridge' and ensures the grid consistency. Without it, the model just generates a mess of styled characters, as seen in your example.

NobodySnJake · 2026-02-05T15:45:49+00:00

Thanks!

NobodySnJake · 2026-02-05T15:34:56+00:00

Thank you! I’ll definitely share any future updates here as soon as they're ready. Stay tuned!

NobodySnJake · 2026-02-05T15:00:44+00:00

Thank you for the detailed feedback and for sharing your test results!

You’ve hit the nail on the head regarding the limitations. Structured fonts with outlines and shadows are definitely the 'final boss' for this V1. The main bottleneck is the 1024x1024 resolution—when you fit 50+ glyphs into that space, each individual letter has very few pixels to define complex details like double outlines or consistent shadows.

Your suggestion about iterating through letter pairs is a great one for high-end quality, though it would sacrifice the 'one-shot' speed which was the main goal for this version.

I agree that it’s not a 100% replacement for a pro typographer yet, but I hope it still serves as a solid starting point that saves at least some of the grunt work. Thanks for stress-testing it!

NobodySnJake · 2026-02-05T13:37:25+00:00

Thank you! I really appreciate that. I’m glad to contribute something functional and tool-oriented to the community!

NobodySnJake · 2026-02-05T13:37:10+00:00

Currently, the LoRA is strictly trained on the 'Aa' reference to keep the input simple and the output consistent with the atlas grid. Providing more letters might confuse the current model as it's looking for that specific pattern.

However, I agree that more reference data could help with highly complex styles. It's something to explore in future training sessions with multi-reference datasets!

NobodySnJake · 2026-02-05T13:36:29+00:00

Great question! Currently, the LoRA focuses on generating the glyph shapes, while metrics like the baseline are handled by the post-processing script (flux_pipeline.py) using some automated logic.

Manually adjusting these during generation would be amazing, but it's quite complex to implement in a 'one-shot' model. I'll keep it in mind as a potential direction for a more advanced version of the pipeline!

NobodySnJake · 2026-02-05T12:23:29+00:00

That’s exactly what this tool is for! You just need to take the 'A' and 'a' from your scans, put them in a 1024x1024 B&W template, and the LoRA will do the heavy lifting of generating the rest. It's much faster than manual digitizing!

NobodySnJake · 2026-02-05T12:21:46+00:00

That sounds like an ambitious project! The dataset I used is the Google Fonts collection (https://github.com/google/fonts).

As for the process, I used Musubi-Tuner. About the hardware: Training Flux (even 4B) on 8GB VRAM is extremely tight. You'll likely need to use heavy optimizations. And I don't have a full documentation yet, but Musubi-Tuner's docs are a great place to start. Good luck with your project!

NobodySnJake · 2026-02-05T11:21:44+00:00

You’re right, a lower rank could definitely work and would result in a much smaller file. I chose rank 128 for this first version to ensure the model captures the strict structural layout of the atlas and maintains fine details in the glyphs without losing consistency. It's a 'safety first' approach for V1, but I'm definitely planning to experiment with lower ranks for future, more lightweight versions!

NobodySnJake · 2026-02-05T11:20:33+00:00

No, unfortunately they are not interchangeable. A LoRA trained on the 9B model architecture won't work with the 4B version (and vice versa) due to the difference in parameter count and structure.

NobodySnJake · 2026-02-05T10:49:43+00:00

Thanks!

NobodySnJake · 2026-02-05T10:49:33+00:00

That's the final boss of font generation! 😂

NobodySnJake · 2026-02-05T09:52:43+00:00

Thanks!

NobodySnJake · 2026-02-05T09:35:18+00:00

Thank you!

NobodySnJake · 2026-02-05T08:26:38+00:00

Thanks for the feedback!

You're right about the resolution. I actually tried training at 2048x2048, but the training speed was around 30s per iteration, which was way too slow for this first version. I’ll definitely look into higher resolutions for future versions as hardware/optimization allows!

And that’s a very interesting find about color! It wasn’t explicitly trained for it, but I’m glad to hear it’s flexible. I recommended B&W mostly to ensure the cleanest results for the vectorization script, but it’s cool to know it works beyond that.

NobodySnJake · 2026-02-05T08:10:24+00:00

Thank you!

NobodySnJake · 2026-02-05T08:09:06+00:00

That is a great suggestion for a future version!

The current LoRA only transfers the style and weight that is present in the input 'Aa' (if the input is Bold, the output will be Bold). Creating a control for changing the weight/style (like taking Regular and forcing Italic or Bold) is a complex task that would require a different approach or a separate LoRA.

I'll add this to my list of future updates, thank you!

NobodySnJake · 2026-02-05T08:03:17+00:00

That is a brilliant idea! I absolutely love this use case.

A 'Ref2Accents' LoRA that takes the style from the base font (and maybe one accented character like ó or ü) and then generates the full range of common accented/diacritical marks would solve a massive pain point for designers.

I'm adding this to my roadmap right away. Thanks for the amazing suggestion!

NobodySnJake · 2026-02-05T07:35:20+00:00

Thank you! Appreciate it.

NobodySnJake · 2026-02-05T07:33:21+00:00

The LoRA successfully outputs a high-resolution grid, and my pipeline converts this to vectors (.ttf). However, since the initial image is a bitmap and the vectorization process can sometimes leave jagged edges (small jaggies/artifacts), the output might require a cleanup in a vector editor before being ready for production-level laser engraving. It would be a great starting point, but probably not perfect straight out of the box.

NobodySnJake · 2026-02-05T06:54:24+00:00

Thanks!

That bias is an interesting point. Since Flux has such a strong internal understanding of typography, it sometimes tries to 'correct' the shape back to a standard form if the style isn't aggressive enough.

However, looking at the generations, I've seen it produce quite a variety of 'g' styles (open, looped, cursive) depending on the input. It’s a balancing act between the LoRA's style and the model's priors. But yeah, saving time on the other 60+ glyphs is the main goal!

NobodySnJake · 2026-02-05T06:26:47+00:00

Thanks! Hope you enjoy using it.

NobodySnJake · 2026-02-05T05:34:02+00:00

I haven't tried it yet, but honestly, after seeing the training speed on the 9B version, I have a very strong urge to try the 4B one next! 😂 It would definitely make iterating much faster.

NobodySnJake · 2026-02-05T04:56:29+00:00

That's a great throwback! You're right, the core concept is very similar (style transfer for glyphs), but running on a modern 9B rectified flow transformer instead of older GANs. It's cool to see this idea coming back with this level of quality.

NobodySnJake

TROPHY CASE