Need help for finetuning script

[Harsh-Kesharwani]

@Zheng-Chong I have build stable-diffusion model from scratch, on top of it i made some changes to obtained CatVTON model architecture. Also successfully loaded pre-trained weights (sd-v-1.5-inpainting). can you help me to finetune this model from scratch on viton-hd dataset. can you provide some reference which i should follow to finetune this model build up from scratch.

github link: https://github.com/Harsh-Kesharwani/stable-diffusion/tree/CatVTON?tab=readme-ov-file
can you please check pipeline.py file (inference code is correct or not)?

[Zheng-Chong]

I’ve already checked your pipeline.py file and there doesn’t seem to be any obvious issue. Perhaps you could describe the specific problem you’re encountering in more detail.

[Harsh-Kesharwani]

@Zheng-Chong can you please tell me what is input to unet model in forward training process for mask based model.
Input 1: (masked person + garment) concat (mask + zero padding)(i.e Mi as refer in paper) concat (noise added to (unmasked person + garment)) and loss computation = MSE(unet(input1) - noise)
Input 2: (masked person + garment) concat (mask + zero padding)(i.e Mi as refer in paper) concat (noise added to (masked person + garment)) and loss computation = MSE(unet(input2) - noise)

can you please tell me which input should be correct in forward process while training masked based model.

[Zheng-Chong]

Input 1

[Harsh-Kesharwani]

@Zheng-Chong does mask-based training is only performed to generate datasets for mask free training ? can i directly train maskfree model on VITONHD dataset ? is training input for maskfree model : noise added to (person with target garment) + concat (person with random garment, target garment)