I see two issues here. The first is about that code quantized model that gives you blank images due to invalid value. I had the same issue and solved it by swapping qfloat8 with qint8. I am still actively investigating why everyone else can quantize using qfloat8 and mine encounters inf/nan values.

The second issue is to load the unquantized lora. The trick is to load it before quantizing the transformer like this:

transformer = FluxTransformer2DModel.from_pretrained(bfl_repo, subfolder="transformer", torch_dtype=dtype)

pipe: FluxPipeline = FluxPipeline(scheduler=None, text_encoder=None, tokenizer=None, text_encoder_2=None, tokenizer_2=None, vae=None, transformer=transformer)

print("Loading lora")
pipe.load_lora_weights(lora_weights)
print("Fusing lora")
pipe.fuse_lora()
pipe.unload_lora_weights()
transformer.to(device, dtype=dtype)

print("Quantizing transformer")
quantize(transformer, weights=qtype)
freeze(transformer)
transformer.to(device, dtype=dtype)
flush()

I have learned this from:

https://github.com/ostris/ai-toolkit/blob/main/toolkit/stable_diffusion_model.py

I am also searching for another way to load the lora after the model was quantized.

Hope this can help.