Help updating/recreating DescriptorLayout with DescriptorBuffer

Hello. So, I have managed to make DescriptorBuffer work. I am allocating about 100K descriptors for textures but I had this idea that, if for some reason we reach that amount, I would allocate 100K more. So, I was trying to test the recreation of the DescriptorBuffer or the Layout.
In a function called IncreaseDescriptorCount, I allocate a bigger buffer, recreate all the SetLayouts and then copy the old data from the old buffer to the new one using memcpy. And then I recreate all the pipeline objects and the pipelineLayout.

Right now, I am using 2 sets in a single DescriptorBuffer. The second one has the Fragment Shader stuff and the first one has the Vertex/Mesh shader stuff.
It renders fine, then when I increase the count, every model becomes black. But the models move and render fine. I can add new models or remove them. But the textures stop working. So, it seems like the first set which I am not modifying is still working fine but the second one which I am modifying isn't.

I am not getting any validation errors and usually I can debug fine with Nvidia Nsight, but after I have increased the descriptor count and recreated the DescriptorBuffer, NSight dies and my graphics driver crashes and my 3D app gets stuck and keeps leaking memory. I guess this is because NSight tries to inject into it and since NSight dies something goes wrong with it and the CommandBuffers don't get reset. Oh and only when I open it with NSight and NSight dies, I get flooded with Validation errors that the CommandBuffer is being reset when it is still in use.

I guess I don't really need this feature. But I feel like it is only not working because I am doing something silly. So, thought I should ask here. Thank you.

Edit: Emm, so turns out I was memcpy ing the old descriptor at the wrong offset. The issue was that I had assumed that the different bindings would be ordered in memory as their binding orders. But not really, for example my textures were at the Binding 2 and the memory offset was 0 or 256B (since it is the second set). But the memory offset for Binding 1 was 320B. So, the texture descriptors were being copied correctly but the rest weren't. My bad, probably should have debugged some more before making this post.

If anyone's interested, here is the repo razerx100/Terra: Renderer module based on Vulkan. (github.com) , hopefully I have finally figured it out.