Inconsistent Head/Body Matching and Split Encounters

Support
,
1

In which Wildbook did the issue occur?
SeadragonSearch

What operating system were you using? (eg. MacOS 10.15.3)
MacOS Catalina 10.15.7

What web browser were you using? (eg. Chrome 79)
Chrome Version 86.0.4240.198

What is your role on the site? (admin, researcher, etc)
Admin

What happened?

This is not preventing me from making matches, but I wanted to let you guys know since it seems the system is not functioning in a consistent way.

I’ve noticed this week that some encounters match only on the head, some match for head and body on one photo (left or right side) but not the other, some match for head and body on both photos (left and right sides), and in some cases photos that were submitted together are split into different encounters.

(I’m assuming that when 2 sets of match results for one photo are returned, it’s trying to match for both body and head, even though the photos in these examples are head shots only.)

Here is an example for each scenario with similar photos:

1 set of matching results per left and right photos:
(assuming matching on the head only)
https://wildbook.dragonsearch.org/encounters/encounter.jsp?number=4f221aba-b53b-4f07-a01a-0922ee74bc3d

2 sets of match results for both left and right sides:
(assuming matching on head and body both)
https://wildbook.dragonsearch.org/encounters/encounter.jsp?number=4b1ac4ea-b4ed-4170-99a0-491d81c08daa

2 sets of match results for right side but only 1 set of match results for left side:
https://wildbook.dragonsearch.org/encounters/encounter.jsp?number=88681051-a3a6-4ad2-99e6-2f165505873c

Photos submitted in 1 encounter but split into 2 separate encounters:
https://wildbook.dragonsearch.org/encounters/encounter.jsp?number=f6640b1b-f609-41fc-8988-b3b60bdf5727
https://wildbook.dragonsearch.org/encounters/encounter.jsp?number=be2f2fa4-b8c3-4c2b-9bf7-87d1bc8e1588

If you want to see more examples I can provide them, or you can just click through encounters of NSW-0106; there’s lots of examples there!

What did you expect to happen?

Since all of the photos in these examples are very similar head shots, I thought the system would handle them all in a similar manner (either consistently include body matching, or consistently not include it). I also didn’t expect photos from a single encounter to be split into multiple, though this has been happening recently, and I’ve posted about it in another thread.

What are some steps we could take to reproduce the issue?

You could try submitting seadragon photos and see what happens, but as I mentioned the results are unpredictable. Let me know if you need more info!

If this is a bulk import report, send the spreadsheet to services@wildme.org with the email subject line matching your bug report

2

I wanted to add this link as well because I think it’s a helpful way to see how many split encounters there are:

https://wildbook.dragonsearch.org/individuals.jsp?id=0067c667-80bd-4a6d-bb6b-5f85626d2b64

If you scroll down to the table of encounters for NSW-0106, any encounter date that appears twice is one where the photos were submitted together but then got split up by the system. Now that I’m seeing how often this is happening, it’s becoming more concerning. It gives an inaccurate picture of how many encounters there are for an individual because it essentially doubles them, but I’m hesitant to delete the split pieces since I don’t know how that might impact the system and future matching.

Thanks for your help with this, Wild Me team!

@NeridaWilson- I’m not sure how much crossover there is between these problems and the ones you’ve been having, so I posted separately, but wanted to keep you in the loop!

-Chrissy

3

Hi Chrissy.

Thanks for providing many examples and links for this post, it really helps the diagnoses process.

It seems there are a couple things happening here that are related.

For your first example “1 set of matching results per left and right photos”, Looking at the annotations created it seems none classified as body were returned from the detection model. You only see one set of results because both of these images only have one annotation- the head. Since these are cropped head shots the model still found the desired matching area, but it’s slightly sub ideal because we would hope a body annotation would always exist and contain the head annotation.

In the second example “2 sets of match results for both left and right sides:” we see body and head annotations, and results for each. This is the better case. Even with cropped images both the head and the slightly larger cropped off body annotation are found. Even though in the cropped images the bounded areas are very similar you can still run matching on each if you wish.

In the third example “Photos submitted in 1 encounter but split into 2 separate encounters” you have the first image with the good case- two separate annotations on the same encounter, but the second image was split off into a new encounter.

This is because both of the annotations in this image were identified as sea dragon head, and it made a new encounter for what it thought was a second individual so as not to have an encounter name conflict.

So this all boils down to Wildbook correctly handling inconsistent responses from the detection model, and largely because of cropping. The problems we see are missing bodies, and doubled heads for images where most of the information that identifies an annotation as ‘body’ isn’t actually present in the image.

Since the logic all seems to be working correctly the way to avoid these effects is to reduce image cropping if possible. If it is not possible and you expect to be working with many more heavily cropped images the only way to change the results is to try retraining the detection model to include more images like this- to teach the model to recognize and classify the body when a very significant portion is obscured.

Hope this helps, sorry it isn’t quite a fix.

1 Like
4

Hi @colin ~

I just realized I never replied to your reply! Thanks so much for this explanation- super helpful to understand what’s going on. We do have a photographer who has a large historical collection of photos that I’ve been chipping away at submitting and processing, and all of his images are close-ups, I guess because of the camera equipment he used to use. So there will be more of these cropped types of photos coming through the system, but mainly from this one photographer.

I’ve been working on submitting a new batch and was reminded of this issue, as it’s happening again with about half of the encounters, which is to be expected given your explanation. I just wanted to confirm that it’s okay for me to delete the bits that have been split off so as not to artificially inflate the overall number of encounters? Just want to make sure I won’t cause any weird ripple effects by doing that :slight_smile:

Thanks again!
-Chrissy

1 Like
5

Hi Chrissy,

Yes, that will work fine. For the images split off into two encounters and two head annotations you
can delete whichever one seems to contain less of the relevant matching area. Since these are already well cropped, it probably doesn’t matter all that much which one.

Thanks for following up!

1 Like
6

Sounds good, thanks @colin ! :slight_smile: