Might I ask you if will be possible to use volume crop changing the spacing scale of your 1268x1454x1064 volume to 0.9, and test this way on the 6900 XT.
This since seems apart the GL_MAX_3D_TEXTURE_SIZE = 2048 topic might be some wrong shared memory management.
Looking at your numbers and seeing the GPU memory usage on my side I got the idea to verify why in some cases you have problems even if the texture resolution is below 2048, and created this table
Res | Hao_Li disk | Hao_Li results | VRAM needed (GByte) |
---|---|---|---|
2367x1784x942 | 7G | No | 29.6 |
1950x1830x1600 | 11G | Wrong stitching | 42.5 |
885x1300x1230 | 5G | Yes | 10.5 |
2125x2510x938 | 10G | No | 37.3 |
1937x1829x951 | 6.5G | Wrong stitching | 25.0 |
2836x1948x2664 | 14G | No | 109.7 |
1928x1927x937 | 6.8G | Wrong stitching | 25.9 |
2098x1919x936 | 7G | No | 28.1 |
1268x1454x1064 | 3G | Yes | 14.6 |
2572x1837x951 | 8G | No | 33.5 |
Where the VRAM needed is calculated considering 16bit (2 byte) resolution at every index, so 8 byte per voxel.
Appear to me you got problems not only if the slice number is more than 2048, but also if the needed VRAM is greather than your GPU VRAM, 16GB in your case.
When both the conditions are met (no more than 2048 slices and no more than 16GB VRAM required) you have no problems (two cases).
So upscaling 10% that volume create a new one requiring more than 16GB and if my hypotesys is true you will need to start having problems even if the slice are less than 2048.