It's basically a trade-off, you want more detail or you want something closer to the real person.

Lora weightage increasing from 0.2, 0.4, 0.6, 0.8