What happened?
I’ve run matches for 6 ID’d wild dog individuals using a left and a right profile for each. In each case, the match results are of other ID’d wild dogs despite there being thousands of unassigned wild dogs in the database. Trying to get more match results by increasing the # still only returns other ID’d pics, not unassigned ones. Only 2 match results overall were from unassigned encounters but 1 of those was actually an ID photo (I can tell bec of the file name) and the other is a perfect matched pose and viewpoint for the target image - in other words, it could be used as a perfect ID pic. Also interestingly, both of these exact same unassigned encounters are presented as the only 2 unassigned match results for both Redding & Escondida.
Note: switching between Individual scores and Image scores doesn’t improve the results.
What did you expect to happen? I expected more match results to come from the unassigned pool.
This explains the low percentage of match results when run for unassigned encounters that we’ve done periodically to date. I just ran 3 randomly selected unassigned encounters from a randomly selected sighting and got the following results:
Hi @tanyastere , we ran into another weird problem with the AWD matches. I’m not sure if it’s related to this current ML issue or something different but I thought I’d add it here bec I think it might be.
Link for issues 1, 2 & 3 below: Wildbook for Carnivores
Issue 1: Running matches on a puppy produced mostly puppies in the match results (expected) but not ones that looked particularly similar to the target.
Issue 2: Within those same puppy match results, there were some opposing viewpoints (the target is a right viewpoint and several of the matches were left viewpoints)
Issue 3: Match #1 is the opposing viewpoint image from the same encounter record (note: these were originally uploaded by JH and so there are frequently multiple images in a single encounter record assigned to a particular ID). Note both these annotations do not appear to have been assigned a viewpoint, which is likely part of the problem, but why don’t they have viewpoints?
Link for issue 4 below: Wildbook for Carnivores
Issue 4: matches 4 & 6 are right viewpoints being offered as matches to the target left viewpoint. Looking at the encounter records for 4 & 6 I can see that they’ve been incorrectly labelled as leftback viewpoints - so that likely explains how they made it into the proposed match list but why were they so poorly id’d as left when they are very clearly and distinctly right profiles?
Sorry for logging this on a Friday evening; I obviously don’t expect an answer, I just wanted to get it in the ticket.
The researcher was able to confirm that the target image of UOM1501 was a match for the dog in the background of the annotation, which includes 2 dogs as is.
i) = the new annotation for the animal in the pic ID’d by the researcher as UOM1501, as shown in the match result 1 from the match run on the original oversized / 2 dogs, 1 annotation encounter record:
When I ran matching on the new encounter with the smaller annotation, (i) above, the system returned with no match results: Wildbook for Carnivores
I expected that the annotation in encounter i) would return a match to the left viewpoint of UOM1501, since it found that match when matching was run on the inverse - left viewpoint of UOM1501 as the target image.
Not sure if this is helpful or even if this is possibly an expected result? But I thought I should add it to this report just in case.
Unfortunately we can’t confirm that this is a bug rather that just how the system works. We’ve put a lot of eyes on this issue and the consensus is that hotspotter works best matching these dogs with very clear, side-on photos, which is also understandably the criteria used for “ID” pictures. Hotspotter works by finding similar regions (each region is very small) between two pictures, and (to drastically simplify things) basically counting how many of these regions are shared by a candidate match. I suspect it is just able to find significantly more of these regions of interest on the sharp, in-focus ID pictures that naturally have a lot of information. That’s why you see many more results on clear photos than on blurrier ones. And in comparison, the unassigned photos at the end of your post are more difficult to match than the ID and ID-style pictures in the earlier examples.
The fact that there are un-id’ed “ID-style” pictures in the results supports to me that there is not a bug going on in the filtering logic that selects the match-against set, which would be the other potential culprit for what you are describing.
I’m curious if you would see the lower matching accuracy you associate with unid’ed photos on id-quality, yet unassigned photos. Input photo quality can make a big difference in matchability.
Thanks for the detailed reporting, having a lot of links really helps.
-Drew
Regarding your second comment, it sounds like the ID system is working as intended with viewpoints but there are some mislabeled viewpoints, and some that are simply not labeled. This could simply be due to the viewpoint labeler system making an incorrect call, as will happen sometimes. If you believe there are systemic errors (or missing labels) on any particular data set let us know and we can re-run them through detection.
And your final comment I’m afraid also has an unsatisfying answer. Hotspotter is not necessarily symmetrical, meaning if A matches to B, it is not a given that B will match to A, which might be confounded by changing the annotation in between both matches (though I recognize the highlighted relevant region is still in the annotation). These would be especially true if the matching sets are not the same. I would not worry about this as a bug unless you see it systemically; there is sometimes edge-case weirdness with these things.
Re: the ID’d matching looking strange and checking un-ID’d encounters to see what we get, I’ve run a few un-ID’d sightings of dogs that are in packs that have ID’d members in ACW - that is, some encounters should match to some ID’d individuals. I’m seeing more of what I would expect - an occasional ID’d encounter that’s a match and other un-assigned encounters. But the volume of proposed matches is still quite low - 0-4 on average, which is less than I expected but that’s not a scientific expectation by any means.
And while I understand what you’ve explained about why the ID’d images appear to be matching to each other even tho they’re different individuals and not actually a match, it’s disappointing to see as that seems to render the ID images less useful than one would expect, almost not useful at all. Our plan was to run matching on all of the ID’d individuals first, before running through any un-assigned images because we thought that would pick up a big swath of animals from the un-assigned batches, but it doesn’t look like it’ll work that way. So we’ll probably change that approach.
But why aren’t we seeing this problem with cheetahs? They have much less differentiation between them than wild dogs (plain black spot patterns, no marbling of tri-coloured patterns) therefore are much more likely to match to each other, especially with ID’d pics, but we don’t see this same issue when running ID’d cheetah individuals. I would think it would be much worse with cheetahs than with wild dogs but it’s the opposite.
My biggest concern - it worries me a lot that un-assigned images that are not as clear as the ID pics may never get matched to a correct and already ID’d individual via the system, possibly leading a user to assign a new ID, not realizing that the animal already exists in the system. What are your thoughts on that?
We’ll keep an eye on the viewpoints issue; my concern there is that it’s not fixable by the user, like keywords are, and it impacts matching. Is there a way to make it fixable? I.e. if the viewpoint is materially wrong, is there something we can do to fix that? Or do I need to make that a feature request?
I understand and accept that A matching B doesn’t always mean that B will match to A but with my concern above under 1), this additional level of randomness adds to that.
Would really appreciate your thoughts on my concern above at the bottom of 1).
Regarding cheetahs, it’s a bit unintuitive but HotSpotter works best on clear and distinct patterns of local contrast, and it’s worse at looking at larger blocks of color. It was first developed for zebras, which as far as the algorithm is concerned, have more similar types of patterns to cheetahs than wild dogs. Even though cheetahs are quite flexible and move as much as any cat, I expect they more often have clear sections of shared contrast-pattern between photos than wild dogs. The density of spots is like a density of information for matching purposes.
I will see this week if we can’t lower the sensitivity for HotSpotter on wild dogs so that you see more results. It’s currently an unknown whether that will truly increase the accuracy or whether additional results will generally be incorrect.
Maybe a good way to update your planned protocol, if it’s doable, would be to prioritize the clearest photos first, since the quality seems to impact matching so strongly.
Thanks, and I’ll be in touch about the HotSpotter sensitivity.
-Drew
Hi @Drew - honestly wild dogs sure like to make it tough for us! I’m concerned about the same thing as you with changing Hotspotter’s sensitivity for wild dogs - that we don’t know if it will help, hurt or have no impact on the match results.
Unfortunately, it’s not really practical to make the shift you’ve recommended in our approach, assuming I’m understanding it correctly. I think you’re saying we should select the clearest photos from the un-assigned imageset to run matching against first. We have thousands of un-assigned encounters and there’s no easy way to review them ‘en masse’ to cherry pick the best. Also, it doesn’t leave us with a systematic approach through the data so we can’t keep track of which ones we’ve run matching against and which we haven’t. Our first choice approach of running matching against all ID’d individuals first was intended to find all matches of all known dogs so that when we find matches between 2 un-assigned encounters, we can be more sure that it’s a new individual that needs a new ID. For now, we’re continuing with this approach.
thanks and we really appreciate the insights and efforts.
Indeed these dogs make it difficult! We’re glad for the challenge, but it is a challenge!
I have decreased the sensitivity of hotspotter so you should see more results; let us know if that helps and we’re definitely interested if there’s a perceived additional accuracy from more results.
One option to find the best quality unmatched photos would be, on an Encounter search, to click “matching photos/videos” to be presented with a gallery of images you can compare side-by-side. This would be after the process of going through and running the ID’d individuals first, which I think is a great idea.
Hi @MarkF - I sense a lot of cleanup getting done I think things are going okay so should be fine to close this one. I know that there’s been a big shift in the success of matching lately although @PaulK got a very weird detection issue in a recent upload: Wild Dog Detector mislabelling tails as bodies
Thanks Mark! As always, I deeply appreciate your diligence and good humour! (yes, here in Canada, there’s a ‘u’ in humour
I think things are going okay so should be fine to close this one. I know that there’s been a big shift in the success of matching lately although 