Strangely low # of match candidates as basis for match results + the single match for 3 diff. dogs are the same annotation of the same dog (Belarus)

What Wildbook are you working in? ACW

What is the entire URL out of the browser, exactly where the error occurred?
4 URL examples are below, screenshots included.
The researcher is running matches using “My data” as his only checked match criteria. He has 15k+ encounters assigned to him of this species and around 450 marked individuals, the majority of which have right side profiles included. So we would have expected to see more right profiles included in the candidate batch than 2 or 3 and also would not expect to see the same annotation proposed as the only match result for each of the 3 matches run, although if only 2 or 3 candidates were selected by the system to match against, it’s less surprising to see the same one show up in all 4 match results.

Examples below from researcher:

  1. We re-ran the right side of Blackbeard and it is only matched against 3 candidates:
    Wildbook for Carnivores

  2. We ran matches on Camargue right side, and again it only checked among three candidates and it pulled up as potential match Belarus. Interestingly, the match result (Belarus) is the exact same match that was offered as match for Blackbeard (above):
    Wildbook for Carnivores

  3. Matches for the right side of Caftan, also only "against two candidates" and it is again offering Belarus as the only potential match:
    Wildbook for Carnivores

  4. Cascade - all viewpoints run through matching (left, right & hind), all produced only Belarus as a proposed match.
    The right profile proposed match (against 3 candidates) is the same Belarus right viewpoint in the other examples here: Wildbook for Carnivores
    Cascade left viewpoint match results (against 2 candidates): Wildbook for Carnivores
    Cascade hind viewpoint match results (against 2 candidates): Wildbook for Carnivores

Blackbeard:
image

Camargue:
image

Caftan:
image

Cascade:

Another note - the hotspot coverage on each of the above proposed matches looks more random than I’m used to seeing and doesn’t appear to match between target & match hotspots other than in a more generalized way in the areas where it seems to be somewhat/possibly similar. Not sure if this is expected either but it’s another element of these match results that appears odd and not similar to my experience with hotspot inspections in the system in general.

Thanks
Maureen

cc: @PaulK

Another example that is similar but not the same -
I logged in as admin and ran matching for this encounter (hind of Bongwe):
http://www.africancarnivore.wildbook.org/encounters/encounter.jsp?number=59f81c78-5f06-4c50-ad76-0bb1472cd84c
I didn’t select any match criteria, leaving the default criteria as is. I got the following match results -
http://www.africancarnivore.wildbook.org/iaResults.jsp?taskId=3ac19e36-4f2d-47db-8260-251ecc582bda
Note that the above match results show 12 proposed matches from a batch of 110 candidates:

The researcher subsequently logged in as himself, researcher rights, and re-ran matching for the same hind annotation. He did select “My data” in the match criteria and got this result:
http://www.africancarnivore.wildbook.org/iaResults.jsp?taskId=cef5dce2-a813-4ae0-bf16-40a72b770e13
Oddly, he ran against a smaller subset of data than I did but his results showed a batch of 4113 candidates as opposed to the 110 in my results. Despite this larger batch of candidates run against, his match results returned 0 potential matches:


He re-ran matching twice and got 0 potential matches both times.

It seems odd to me that the # of candidates is lower for my match results, running against the full database, than for his match results, running against a subset of the database. I also find it strange that, despite the above, his candidate batch of 4113 returned no results vs my candidate batch of 110 returning 12 proposed matches.

Not sure if this is entirely expected and I’m just not understanding why and/or if it’s related to the issue reported above. But I thought I should include this here in case it helps.

thanks!
Maureen

cc: @PaulK

Hi, our researcher has run some additional matching and has been tracking the “against x# of candidates” in the match results. I think it may be additional info to assist in the investigation of the above issue so I’m adding it here. We find that the # of candidates run against in each case is odd - why more candidates instead of less, when “my data” is selected as criteria? And then odd to see more proposed match results when run against fewer candidates in the database, over the larger volume of candidates stated in the “My data” filter?

Here’s what the researcher sent me:

I started 6 matches:

  1. I ran a match on the left side and I checked the box with the option “My data”. It checks against 3567 candidates and offers only one match. The match is correct and the encounter “BPC_OW_FLDR246_…” is now assigned to Etna.
    Wildbook for Carnivores

  2. I ran a match on the left side and I DID NOT check the box with the option “My data”. It searches against 131 candidates and offers 12 possible matches. All are wrong and all are already named dogs.

http://www.africancarnivore.wildbook.org/iaResults.jsp?taskId=ee099a10-4421-4902-9a82-f0b3e7569c72

  1. I ran a match on the right side and I checked the box with the option “My data”. It checks against 3567 candidates and offers no possible matches. I find it veeeeery weird that it searches against the same number of candidates as the left side under point 1)

http://www.africancarnivore.wildbook.org/iaResults.jsp?taskId=f810e807-902e-43de-9afd-5d8fceac8a2f

  1. I ran a match on the right side and I DID NOT check the box with the option “My data”. It checks against 182 candidates and offers 12 possible matches. All are wrong and all are already named dogs.

http://www.africancarnivore.wildbook.org/iaResults.jsp?taskId=0b89ce3a-fdf3-4061-be22-54a94159153a

  1. I ran a match on the hind side and I checked the box with the option “My data”. It checks against 3567 candidates and offers no possible matches. 3567 is the same number of candidates as under point 1) and 3).

http://www.africancarnivore.wildbook.org/iaResults.jsp?taskId=e9a8c955-f85e-483d-8b3a-97407b3c8985

  1. I ran a match on the hind side and I DID NOT check the box with the option “My data”. It checks against 181 candidates and offers 2 possible matches, both wrong and both already named dogs.

http://www.africancarnivore.wildbook.org/iaResults.jsp?taskId=c5f1a241-d15b-4c58-a9e4-d14cda8f46eb

Thanks
Maureen

Hello,

There is a fix deployed for this behavior that is now live on ACW. When selecting ‘My Data’ for determining a matching set we will now check against the submitterID for an encounter instead of the submitter list. If the submitter list did not contain your username, even if you had ownership of an encounter it could return zero results for a matching set. Discrepancies here could have produced an artificially small or empty list of matching candidates.

The identification system does a fallback comparison if provided no candidates where it compares against everything of the appropriate species and viewpoint in the system. This is why the results could be large and not limited by location or user- the ‘My Data’ query inaccurately turned up nothing. This also explains how the large candidate set number could be the same for multiple examples you provided.

We think that this is the reason choosing ‘My Data’ for a matching set could have unpredictably resulted in incorrect matching sets.

The internal issue ticket closed by this is WB-1626.

Thanks!

1 Like

Hi @colin, I’m not sure I’m understanding what’s been fixed bec I don’t know what the “submitter list” is and where that is as it relates to an annotation or encounter?

Also, I’m not sure we’re referring to the same issue?
If you look at the Bongwe example above, we found more candidates when My data was selected (4113 candidates) versus 110 candidates when I ran matching against the whole database (no match criteria selected), under my admin account.

Also, what’s the reason for the same annotation of the same individual to be offered as the only match results for 4 different individuals? That seems like a strange coincidence?

That said, I’m not sure everything is working as it should even with the fix that’s just been applied. We’ve re-run the Blackbeard right viewpoint encounter above (https://africancarnivore.wildbook.org/encounters/encounter.jsp?number=4c1894ea-e919-41af-a1f9-c5145fc882a0) - once with My data applied as a filter (under the researcher’s user ID) and once without any match filters applied. The results were as follows:
With My data selected: 360 candidates with no matches found:

With no match criteria selected: 359 candidates with 1 proposed match:

I would have expected the My data filter to return less candidates than the no filter results. What am I misunderstanding?

thanks
Maureen

Hello,

I’ve run the match again without a submitter based limiter and it says 360 candidates for me:

Also no match as you saw.

Since it is only a difference of one and I did not see this effect I suspect some stale caching either in the database or the browser.

I also looked for Lycaon Pictus in the location ID Botswana. That is the location ID for this encounter, although I see many more for the broader location ID ‘Africa - South’. There are 716 suitable encounters for that query, and 714 of them belong to Gabriele. It is entirely possible that you would return nearly the same or exactly the same matching set whether you set “My Data” or not for this matching job.

Also, I’m not sure we’re referring to the same issue?
If you look at the Bongwe example above, we found more candidates when My data was selected (4113 candidates) versus 110 candidates when I ran matching against the whole database (no match criteria selected), under my admin account.

I think the ‘My Data’ query returned zero candidates for this specific example. If it returns zero candidates and sends the job to image analysis anyway, then image analysis gets to decide and sometimes the “Just Try Anyway” matching set can be quite large.

Also, what’s the reason for the same annotation of the same individual to be offered as the only match results for 4 different individuals? That seems like a strange coincidence?

It’s a little weird but not unheard of. I’d guess that out of those extremely small matching sets Belarus is truly the best candidate. His encounter has a very high quality images of left and right viewpoints.

Thanks, hope that helps.

Hi @Colin - @jason, Paul and I just had a chat about this and we agreed that adding code to tell the system to match any IA class that starts with “wild_dog_” should be matched against all other “wild_dog_”.

For any IA class that starts with “wild_dog+”, they should only be matched against other “wild_dog+” IA classes in the future; however, at this time these IA classes with the “+” sign are all tail parts and these are currently set to “false” or to not be matchable. For now, we don’t have a need to change this to allowing tails to be matchable to each other, although we may in the future. But for now, we can ignore the matching between tail classes because of this.

I hope that makes sense! Thanks to both of you for working on this. We like the solution!

best,
Maureen & @PaulK

@ACWadmin1 @PaulK

This has been implemented, and you should see matching of wild dogs regardless of coloration.

Thanks,
Jason

2 Likes