Skip to content

LoRA load issue #11659

Open
Open
@D1-3105

Description

@D1-3105

Describe the bug

FluxPipeline fails to load LoRA weights (https://civitai.com/models/876464):

Reproduction

from diffusers import FluxPipeline
from torch import bfloat16

prompt_from_repo = "Beautiful female woodelf in shiny iron and silver armor, full body view, full body shot, feet and arms seen in the view, hyper photorealistic, ultra photorealistic, looking straight into camera, Elf, pointy ears, photorealistic, fighting pose, shoulder length wavy red hair, full body, athletic figure, beautiful green eyes, 8k, ultra realistic, hyper realistic, highly detailed, daylight, fantasy style, high contrast, colorful polychromatic, (Intricately Designed face: 1.3), dramatic lighting, wet and sweaty skin, dirty skin, Cinematic colorful lighting, realistic body and face, Photograph Taken on Nikon D750, Intricate, Elegant, Digital Illustration, Scenic, Hyper-Realistic, Unreal Engine, CryEngine, Octane Render, Artgerm, WLOP, Greg Rutkowski, 8k ultra high resolution concept art, hyper-defined, sharp focus, echoing, (hyperdetailed body and face: 1.3), asymmetric balance, Stunning and amazing.,Acrylian4,Lineart style:Charcoal lineart and white chalk lineart,Tannis, Tanis, Esmeralda , piercing multicolor colorful eye color,black ink lineart, dynamic pose, Amazing artwork, a Masterpiece painting"
negative_prompt_from_repo = "score_6,score_5,score_4, , poor artist, bad artist, ugly, poorly drawn, poorly detailed, warped, morphed, uncanny, horror, disfigured, score_4, score_5, score_6, source_furry, wrong hand, bad hands, bad anatomy, fewer digits, bad perspective, bad proportions, bad arm, extra arms, cross-eyed, logo, censored, blurry, lowres, artistic error, watermark, jaggy lines, deformed, monochrome, bad eyes embedding:EasyNegative"
lora_id = "D1-3105/lora_876464__issue11659__DEV-2308"
model_id = "black-forest-labs/FLUX.1-dev"

pipeline = FluxPipeline.from_pretrained(model_id, cache_dir="train_model/huggingface", torch_dtype=bfloat16).to("cuda")

pipeline.load_lora_weights(lora_id)
assert pipeline.get_active_adapters()
image = pipeline(
    prompt=prompt_from_repo,
    negative_prompt=negative_prompt_from_repo,
    width = 768,
    height = 1152,
    seed=442779123,
    steps=25,
    guidance_scale=25,
).images[0]

Logs

No LoRA keys associated to FluxTransformer2DModel found with the prefix='transformer'. This is safe to ignore if LoRA state dict didn't originally have any FluxTransformer2DModel related params. You can also try specifying `prefix=None` to resolve the warning. Otherwise, open an issue if you think it's unexpected: https://github.com/huggingface/diffusers/issues/new
No LoRA keys associated to CLIPTextModel found with the prefix='text_encoder'. This is safe to ignore if LoRA state dict didn't originally have any CLIPTextModel related params. You can also try specifying `prefix=None` to resolve the warning. Otherwise, open an issue if you think it's unexpected: https://github.com/huggingface/diffusers/issues/new

System Info

diffusers==0.33.1

Who can help?

@sayakpaul

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions