Help Needed! Terabytes of Proprietary Image Data, but Training isn't Going Well.

I'm exploring how well stable diffusion can reproduce photogrammetry surface scan data given the imagery in my database. Basically, I want to create my own LoRAs of textures for vfx and games. Lots of folks are doing this with stylized low frequency data, but ours is real data gathered using high resolution cameras and equipment and postprocessed for quality. As a result, the level of detail is very high, but achieving that level of quality is proving difficult.

I'm using RTX 3090s to train, for now.

My Training Approach

  • Our color textures are 8192x8192. My idea is to downsample to 4096x4096 and create 1024x1024 tiles, which result in 16 images per surface. I'm trying to strike a balance between image size and feature distribution so the model can learn what it needs. I'm only training one LoRA per surface so the data is very homogenous
  • The prompt isn't long. I'll start it with a custom word following by a short description. For example, "msurf dusty small floor tile shards on concrete"
  • I'll load up kohya_ss and set up the project. 50 epochs, 100 steps, 10-40 repeats. Adafactor optimizer, 0.0002 learning rate, batches 2. Truthfully, I'm not certain what to do here. Youtube doesn't help.
  • I'm using SDXL base model as I don't want to associate my data with illicit imagery.
  • 4 hours later, inferencing doesn't look too bad, but it's just off. The last several samples it renders during training (I sample each epoch) appears to be overfitting with blotchiness. But, slight distortions are apparent even with earlier checkpoints.
  • I'll adjust the epochs, the steps, the repeats, but in the end I still can't get that grit or level of sharp detail that the ground truth images have. The model gets the structure ok, but those high frequencies are just missing.

Initially I thought to just length the training time but I think it's overfitting as I mentioned earlier. But, is it even possible to get that fine level of detail I'm after?

Thanks for your help and suggestions!