SlicerMorph Photogrammetry photo masking issue

Support
,
1

I am trying the Slicer Photogrammetry with a student. The photos were taken by a cell phone at a single perspective. However, the masking behaved very weirdly.

image
image970×288 33 KB

image
image1202×830 88.5 KB

I’m also not very sure about the orientations. The photos were rotated horizontally in Slicer, and the output masks were also horizontal. The original photos were still vertical.

image
image1479×708 111 KB

Perhaps due to the masking issue, the reconstruction failed at the very beginning.

Is there anyway to add an option to run ODM reconstruction without masking first?

Here are the photos:photos - Google Drive

Thanks!

@muratmaga @oothomas

2

Yes, if that’s the masking result you got, photogrammetry isn’t likely to work on it. I have never seen patterns like that. Are you using the ROI tool to specify what to keep and then fine tune with points? Are there any errors in the console log?

3

Even if you can mask I suspect you are not going to get very good results, since your model shape very uniform and doesn’t have a lot of features. The most important step of photogrammetry is feature extracting and matching. This will actually will help by using something that has more texture than what you have.

For SlicerMorph Photogrammetry, accurate masking is considered crucial because it improves the reconstruction as well as speeds up the process and we have spent a lot of time trying to integrate SAM to achieve that. If you want to skip the masking and upload your photos as is, the best is probably would be launching the ODM on your own (without using the extension).

image

4

I see. I think this model should be fine for regular photogrammetry. I will ask the student to take photos with some background to see if it works. If not, using WebODM might be an option.

5

Also I wonder if the phone is embedding some rotation info in the jpeg header about if the phone was in landscape or portrait mode when the photo was taken and that needs to be taken into account.

2 Likes
6

Are you saying these were already masked and you tried to remask?

7

It makes sense to me, because sometimes my iphone photos just rotated when I imported them into my computer. I feel the output masking may not match the photos, which led to failure.

For example, for some reason, I got a very good masking results, but the task still failed:

image
image1525×926 191 KB

I will try rotating photos in Slicer, or try with a regular camera and also added some background to it.

8

These images looked already processed. So I don’t know how well SAM works on such cases.

image
image1033×1116 49.1 KB

Try on the original ones.

As for your second set of masks, yes, looks like there is some kind of a rotation. The masks should inherit the image information from the photographs, but we never worked with cellphone photos. I will ask @oothomas to take a look. If you can provide the originals as saved by the camera (prior to any processing) would be great.

1 Like
9

I found this interesting thread for similar rotation issue: https://discussions.apple.com/thread/251323176?answerId=252561528022&sortBy=rank#252561528022:~:text=May%201%2C%202020%209,with%20the%20orientation%20tags.

I will ask the student about how the pictures were taken. I feel adding some background might be helpful for masking as well.

10

It would be cool to be able to use a cellphone camera if you can figure out how to account for the orientation tags.

Also I think Murat’s point about the surface of the object needing to have more texture for feature masking makes sense. Maybe try a different toy for testing.

1 Like
11

Hi Chi,

Regarding the first images you shared with the masking issue; I’ve run into this before when the bounding box isn’t tight enough around the objects to be masked. Were you using the include/exclude options at all?

12

I think currently the issue is even when the mask is good, they seem to be rotated with respect to the original image, which might indicate some kind of a portrait/landscape flip in the image headers that we are missing out. but I think we will need the original photos from the phone as well to track where the difference is coming from.

13

Thanks, @oothomas. Yes, I did a big ROI box since the object was not in a stable position. I did not try placing ROI for a single object. I’ll wait for the original photos then give it a try.

14

Since the same ROI is used for all images, setting a very large ROI to accommodate everything gives less than ideal result. If you don’t have too many pictures, you can try to set the ROI manually per photo, which is obviously a bit tedious.

Alternatively, there is ClusterPhotos which sort of tries re-organizes your photos into sets with similar orientation to deal with subject moving in the picture frame. You can also give that a try…

1 Like
15

With the help of ChatGPT, yes, iphone would add a transpose flag when the photos are taken in portrait mode. Slicer is not a photo-editing app, so it’ll read raw images without EXIF data and display it in landscape orientation.

This is a simple script using Pillow to transpose the image to counter the flag:

from PIL import Image, ImageOps
img = Image.open("/home/zhang/Downloads/Test_Out/All Images/IMG_5653.jpg")
exif = img.info.get('exif') 
img_corrected = ImageOps.exif_transpose(img)
img_corrected.save("/home/zhang/Downloads/Test_Out/IMG_5653_corrected.jpg", exif=exif)

Now the image displayed correctly in Slicer:

image
image1344×569 55.1 KB

I will try it soon to see these corrected photos work.

1 Like