[bug]: RuntimeError: "upsample_nearest2d_channels_last" not implemented for 'Half'

[arboeh]

Is there an existing issue for this problem?

• I have searched the existing issues

Operating system

Linux

GPU vendor

None (CPU)

GPU model

AMD Radeon Graphics 512 MB

GPU VRAM

No response

Version number

4.2.9

Browser

Chrome 128.0.6613.138

Python dependencies

{
"accelerate": "0.30.1",
"compel": "2.0.2",
"cuda": null,
"diffusers": "0.27.2",
"numpy": "1.26.4",
"opencv": "4.9.0.80",
"onnx": "1.15.0",
"pillow": "10.4.0",
"python": "3.11.9",
"torch": "2.2.2+cpu",
"torchvision": "0.17.2+cpu",
"transformers": "4.41.1",
"xformers": null
}

What happened

[2024-09-12 08:01:01,348]::[InvokeAI]::ERROR --> Error while invoking session a9ba4712-f7cc-419b-a6f7-fde62d5f0cdc, invocation 08eb64f9-e010-4ad4-a5b9-de858c15645d (l2i): "upsample_nearest2d_channels_last" not implemented for 'Half'
[2024-09-12 08:01:01,348]::[InvokeAI]::ERROR --> Traceback (most recent call last):
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/invokeai/app/services/session_processor/session_processor_default.py", line 129, in run_node
    output = invocation.invoke_internal(context=context, services=self._services)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/invokeai/app/invocations/baseinvocation.py", line 297, in invoke_internal
    return self.invoke(context)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/invokeai/app/invocations/latents_to_image.py", line 110, in invoke
    image = vae.decode(latents, return_dict=False)[0]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper
    return method(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/diffusers/models/autoencoders/autoencoder_kl.py", line 304, in decode
    decoded = self._decode(z).sample
              ^^^^^^^^^^^^^^^
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/diffusers/models/autoencoders/autoencoder_kl.py", line 275, in _decode
    dec = self.decoder(z)
          ^^^^^^^^^^^^^^^
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/diffusers/models/autoencoders/vae.py", line 338, in forward
    sample = up_block(sample, latent_embeds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/diffusers/models/unets/unet_2d_blocks.py", line 2741, in forward
    hidden_states = upsampler(hidden_states)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/diffusers/models/upsampling.py", line 172, in forward
    hidden_states = F.interpolate(hidden_states, scale_factor=2.0, mode="nearest")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arend/InvokeAI/.venv/lib/python3.11/site-packages/torch/nn/functional.py", line 4001, in interpolate
    return torch._C._nn.upsample_nearest2d(input, output_size, scale_factors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: "upsample_nearest2d_channels_last" not implemented for 'Half'

What you expected to happen

Text2Image generation works with VAE FP16 (and any SDXL model).

How to reproduce the problem

Generate text2image with VAE FP16 (and any SDXL model).

Additional context

Text2Image generation with VAE FP16 doesn't work. VAE FP32 works.
madebyollin/sdxl-vae-fp16-fix doesn't work either.
Ubuntu 24.04 LTS in WSL.

Was discussed here also, but without answer: #3477

Discord username

No response

[psychedelicious]

Hi, the linked issue is for a different application, not Invoke. That said, it looks like the root cause is the same - the torch operation required isn't available on the CPU. It's likely that using the FP32 precision VAE is the only option, pending a fix from torch.