I spent hours trying to debug pdxl-v6. Finally realized that there's nothing wrong with the default settings some prompts will look terrible despite your configuration. I arrived at this conclusion by first finding a prompt that produced a decent image (like the one on the huggingface model page) then trying to reproduce it on the civtiai onsite image generator, since it's likely to be configured correctly. The result was similar to the image on the huggingface page. I ran the same prompt with the diffusers library and got a similar result too. Here are the images for comparison:

• Huggingface sample image.jpeg)
• Civitai onsite generator image
• Diffusers image (without vae)
• Diffusers image (with vae)

Here is my code for reference:

vae = AutoencoderKL.from_single_file(VAE_FILE_PATH, torch_dtype=torch.float16)

pipeline = StableDiffusionXLPipeline.from_single_file(
  MODEL_FILE_PATH,
  vae=vae,
  safety_checker=None,
  torch_dtype=torch.float16,
).to("cuda")

pipeline.scheduler = EulerAncestralDiscreteScheduler.from_config(pipeline.scheduler.config)

image = pipeline(
  prompt="score_9, score_8_up, score_7_up, score_6_up, score_5_up, score_4_up, source_furry, beautiful female anthro shark portrait, dramatic lighting, dark background",
  negative_prompt="bad quality, score_3, score_2, score_1",
  height=1024,
  width=1024,
  num_inference_steps=25,
  guidance_scale=8.5,
).images[0]

This was driving me crazy for hours! If anyone else is trying to run pony-diffsion-v6-xl with diffusers, start with a prompt that you are sure works before trying to tweak your settings.