Ryan Dick
9b615e0de7
Fix bugs when switching control layer type. This logic still feels very hacky.
2024-12-17 13:36:11 +00:00
Ryan Dick
a463e97269
Bump FluxControlLoRALoaderInvocation version.
2024-12-17 13:36:10 +00:00
Ryan Dick
b272d46056
Enable ability to control the weight of FLUX Control LoRAs.
2024-12-17 13:36:10 +00:00
Ryan Dick
4d5f74c05b
LoRA refactor to enable FLUX control LoRAs w/ quantized tranformers ( #7446 )
...
## Summary
This PR refactors the LoRA handling code to enable the use of FLUX
control LoRAs on top of quantized transformers.
Changes:
- Renamed a bunch of the model patching utilities to reflect that they
are not LoRA-specific
- Improved the unit test coverage.
- Refactored the handling of 'sidecar' patch layers to make them work
with more layer patch types. (This was necessary to get FLUX control
LoRAs working on top of quantized models.)
- Removed `ONNXModelPatcher`. It is out-of-date and hasn't been used in
a while.
## QA Instructions
I completed the following tests.
**These should be repeated after changing the target branch to main.**
**Due to the large surface area of this PR, reviewers should do
regression tests on a range of LoRA formats. There is a risk of
regression on a specific format that was missed during the
refactoring.**
- [x] FLUX Control LoRA + full FLUX transformer
- [x] FLUX Control LoRA + BnB NF4 quantized transformer
- [x] FLUX Control LoRA + GGUF quantized transformer
- [x] FLUX Control LoRA + non-control LoRA + full FLUX transformer
- [x] FLUX Contro LoRA + non-control LoRA + BnB quantized transformer
- [x] FLUX Control LoRA + non-control LoRA + GGUF quantized transformer
- Test the following cases for regression:
- [x] Misc SD1/SDXL LoRA variants (LoRA, LoKr, IA3)
- [x] FLUX, non-quantized, variety of LoRA formats
- [x] FLUX, quantized, variety of LoRA formats
## Merge Plan
**_Don't merge this PR yet._**
Merge plan:
1. First merge brandon/flux-tools-loras into main
2. Change the target branch of this PR to main
3. Review / test / merge this PR
## Checklist
- [x] _The PR has a short but descriptive title, suitable for a
changelog_
- [x] _Tests added / updated (if applicable)_
- [x] _Documentation added / updated (if applicable)_
- [ ] _Updated `What's New` copy (if doing a release after this PR)_
2024-12-17 08:30:50 -05:00
Ryan Dick
dd09509dbd
Rename ModelPatcher -> LayerPatcher to avoid conflicts with another ModelPatcher definition.
2024-12-17 13:20:19 +00:00
Ryan Dick
7fad4c9491
Rename LoRAModelRaw to ModelPatchRaw.
2024-12-17 13:20:19 +00:00
Ryan Dick
b820862eab
Rename ModelPatcher methods to reflect that they are general model patching methods and are not LoRA-specific.
2024-12-17 13:20:19 +00:00
Ryan Dick
c604a0956e
Rename LoRAPatcher -> ModelPatcher.
2024-12-17 13:20:19 +00:00
Ryan Dick
9369b39a12
Add GGMLTensor op.
2024-12-17 13:20:19 +00:00
Ryan Dick
80f64abd1e
Use a FluxControlLoRALayer when loading FLUX control LoRAs.
2024-12-17 13:20:19 +00:00
Ryan Dick
37e3089457
Push LoRA layer reshaping down into the patch layers and add a new FluxControlLoRALayer type.
2024-12-17 13:20:19 +00:00
Ryan Dick
fe09f2d27a
Move handling of LoRA scale and patch weight down into the layer patch classes.
2024-12-17 13:20:19 +00:00
Ryan Dick
e7e3f7e144
Ensure that patches are on the correct device when used in sidecar wrappers.
2024-12-17 13:20:19 +00:00
Ryan Dick
606d58d7db
Add sidecar wrapper for FLUX RMSNorm layers to support SetParameterLayers used by FLUX structural control LoRAs.
2024-12-17 13:20:19 +00:00
Ryan Dick
c76a448846
Delete old sidecar_layers/ dir.
2024-12-17 13:20:19 +00:00
Ryan Dick
46133b5656
Switch LoRAPatcher to use the new sidecar_wrappers/ rather than sidecar_layers/.
2024-12-17 13:20:19 +00:00
Ryan Dick
ac28370fd2
Break up functions in LoRAPatcher in preparation for more refactoring.
2024-12-17 13:20:19 +00:00
Ryan Dick
1e0552c813
Add optimized implementations for the LinearSidecarWrapper when using LoRALayer or ConcatenatedLoRALayer patch types (since these are the most common).
2024-12-17 13:20:19 +00:00
Ryan Dick
e2451ef5ca
A unit tests for LinearSidecarWrapper (and fix a bug).
2024-12-17 13:20:19 +00:00
Ryan Dick
443d838fd0
Add initial basic implementation of sidecar wrappers.
2024-12-17 13:20:19 +00:00
Ryan Dick
3a8a5442ea
Add basic unit tests for SetParameterLayer.
2024-12-17 13:20:19 +00:00
Ryan Dick
808e3770d3
Remove AnyLoRALayer type definition in favor of using BaseLayerPatch base class.
2024-12-17 13:20:19 +00:00
Ryan Dick
2b441d6a2d
Add BaseLayerPatch ABC to clarify the intended patch interface.
2024-12-17 13:20:19 +00:00
Ryan Dick
58de93a89e
Delete empty file.
2024-12-17 13:20:19 +00:00
Ryan Dick
1eede4315e
Delete ONNXModelPatcher. It is outdated and hasn't been used for a long time.
2024-12-17 13:20:19 +00:00
Ryan Dick
8ea697d733
Mark LoRALayerBase.rank(...) as a private method.
2024-12-17 13:20:19 +00:00
Ryan Dick
693d42661c
Add basic unit tests for LoRALayer.
2024-12-17 13:20:19 +00:00
Ryan Dick
41664f88db
Rename backend/patches/conversions/ to backend/patches/lora_conversions/
2024-12-17 13:20:19 +00:00
Ryan Dick
42f8d6aa11
Rename backend/lora/ to backend/patches
2024-12-17 13:20:19 +00:00
psychedelicious
5f41a69665
feat(ui): prevent invoking when >1 control lora enabled
2024-12-17 07:28:45 -05:00
Ryan Dick
7da90a9b6b
Ensure that model probe does not crash with integer state dict keys.
2024-12-17 07:28:45 -05:00
Ryan Dick
440185cc40
Simplify FLUX control LoRA probing.
2024-12-17 07:28:45 -05:00
Ryan Dick
26edc71268
ruff format
2024-12-17 07:28:45 -05:00
Ryan Dick
a4bed7aee3
Minor tidy of FLUX control LoRA implementation. (mostly documentation)
2024-12-17 07:28:45 -05:00
Ryan Dick
5fcd76a712
Fix frontend FLUX graph construction for FLUX control LoRAs.
2024-12-17 07:28:45 -05:00
Mary Hipp
516ffa641c
add logic to change type to control_lora properly
2024-12-17 07:28:45 -05:00
Ryan Dick
d84adfd39f
Clean up FLUX control LoRA pre-processing logic.
2024-12-17 07:28:45 -05:00
Ryan Dick
ac82f73dbe
Make FluxControlLoRALoaderOutput.control_lora non-optional.
2024-12-17 07:28:45 -05:00
Brandon Rising
70811d0bd0
Remove unexpected artifacts in output images
2024-12-17 07:28:45 -05:00
Mary Hipp
e0344a302c
feat(ui): update FLUX graph building to include control layers with control loras
2024-12-17 07:28:45 -05:00
Mary Hipp
92b0d89b70
(ui): replace logic for controlnet/t2i to include control_loras and display default settings in model manager
2024-12-17 07:28:45 -05:00
Mary Hipp
da213e4638
feat(ui): add control loras to control adapter model options, add default settings for preprocessor in probe
2024-12-17 07:28:45 -05:00
Brandon Rising
246b59f148
Run pnpm fix, regenerate schema
2024-12-17 07:28:45 -05:00
Brandon Rising
046d19446c
Rename Structural Lora to Control Lora
2024-12-17 07:28:45 -05:00
Ryan Dick
040551d4fb
Fixes to get FLUX Control LoRA working.
2024-12-17 07:28:45 -05:00
Brandon Rising
f53da60b84
Lots of updates centered around using the lora patcher rather than changing the modules in the transformer model
2024-12-17 07:28:45 -05:00
Brandon Rising
5a035dd19f
Support bnb quantized nf4 flux models, Use controlnet vae, only support 1 structural lora per transformer. various other refractors and bugfixes
2024-12-17 07:28:45 -05:00
Brandon Rising
f3b253987f
Initial setup for flux tools control loras
2024-12-17 07:28:45 -05:00
psychedelicious
25ff7918e8
chore(ui): knip
2024-12-16 18:57:43 -08:00
psychedelicious
09fc60acb0
feat(ui): show toasts when filter, transform, select or crop fails
2024-12-16 18:57:43 -08:00
