IoT MiewID matches incorrect, only checking against recent encounter submissions

Support
,
1

What Wildbook are you working in?
Internet of Turtles

What is the entire URL out of the browser, exactly where the error occurred?
Just two examples here from Oman:

Can you describe what the issue is you’re experiencing?
Encounters from known individuals do not result in the correct matches when running MiewID (false negatives). Both examples above are encounters from known individuals (1. from one of our most sighted turtles GM043), but the matching page does not include the correct individual. It seems the algorithm is only matching against recent encounter submissions or with a preference to them, not against all previous encounters from the Location ID.

All suggested matches from the top twelve have a score between 0.6-0.8, when previously there was a wider range with distinctly higher score for the correct match using the Hotspotter function.

Similar issues reported form ORP-Laamu and ORP-Lhaviyani accounts for Maldives - I will request further examples to be added to this thread from these locations.
Issue is appearing on researcher accounts, as well as orgAdmin account (ORP).

Additional side note: Encounters for one Location ID are predominantly uploaded from one or two account (e.g., Laamu encounters from ORP-Laamu or ORP-Olhuveli, Lhaviyani from ORP-Lhaviyani etc) and have been in the past as well.

Can you provide steps on how to reproduce what you’re experiencing?
Upload an encounter, wait for automatic detection & matching to process, check suggested matches.

Please let me know if this description needs further clarification.

Best,
Steph

2

Possibly related to All match candidates are from my own submissions

Thanks for the detailed summary, @ORP-StephanieK! I’ll research this along with what I suspect could be a related issue and follow up if I have any questions or updates. I’ll keep an eye out for additional examples from your colleagues.

3

I want to add that match scores are represented differently with each algorithm. In some, the higher the score, the higher the likelihood of a match. In others, it’s the opposite and a low score represents the closest match. What the score represents can also vary from one trained instance of the algorithm to another. In general, we recommend focusing on the ranking instead of the match score.

We’re also in the process of working with a team at CMU (Carnegie Mellon University Research Study: Think Aloud Study to Understand Model Confidence) to make the information on the match page easier to break down (ranking/match score/confidence/etc.)

That said, I do see that GM043 comes in at rank 17 in the first image of your fist encounter link, which I agree should ideally be higher.

2 Likes
4

Hi Anastasia,

Just adding the example submissions from Laamu Atoll, Maldives that Stephanie had requested:

These are two well-known individuals (HK3822 & GR1449) from the area but the match results have been unsuccessful at finding a correct match OR the correct match was 9th on the list.

Thank you for looking into this. Please let me know if you need any other examples.

Julian

5

Thanks, @JoGoodfellow!

We hope to have a resolution soon.

6

Thanks for your patience!

My teammates and I researched this and didn’t find any systemic problems with match results in IoT. AI isn’t perfect and occasionally, known individuals aren’t suggested as possible match candidates 100% of the time. What we’ve observed through testing several matches this week is within the expected range of success.

Correct matches may not always appear in the top 12 so when in doubt, increase the Num Results field on the match page and you may find your candidate with a lower match score (due to available viewpoints to compare against, photo quality, etc.).

@JoGoodfellow Even though HK3822 doesn’t appear as a match candidate until #9, the fact that MiewID matched it to its baby picture is actually pretty impressive!

We did uncover about 3% of turtles (across all species) submitted to IoT that have not completed detection and as a result haven’t been showing up in match results. This may not be a significant enough number to change match rates, but it should help with making sure all available turtles are matchable.

I’m marking this as resolved because matching is verified to be behaving as expected.

7

Okay, I see.

We’ll keep monitoring and will consider the first 24 suggested matches from now on - I really don’t manage to have more reasonably displayed on my screen tbh.

Thank you for checking.

Best,
Steph

1 Like