What operating system were you using? (eg. MacOS 10.15.3)WIN 10
What web browser were you using? (eg. Chrome 79)Chrome latest
What is your role on the site? (admin, researcher, etc)Admin
What happened? Feedback from 1 user and review of another user’s bulk import matching results shows PIE v2 producing garbage match results as well as errors on hyaena data.
What did you expect to happen?
Match results to be useful from PIE V2
What are some steps we could take to reproduce the issue?
Check bulk imports here
Some good news here. This looks more like a hiccup than a PIE model failure. I am re-running your matches above and absolutely getting matches and correct ones (I think).
Here are example links I just ran:
Are all of these failures from the same bulk import? Have you done any curation on that data? One possibility is just to re-run these.
OK, please feel free to ping me if you see these again as they definitely were zero and wrong. I would like to see whether a bug caused them and if we can fix them.
Good morning @jason - this morning I re-uploaded a small file from the original larger bulk import and ran detection & matching on it. The PIE v2 results now have some (low) scores and the inspect button is now available (it wasn’t on the original upload).
I then took one of the encounter records from this newly uploaded & IA-processed bulk import and started a new match to see if the manual results would be different from the bulk import match results. They were identical. So I think it’s working properly now.
So I think it was a blip caused somewhere in the bulk import processing on that day last week when 2 different uploads from 2 different users had the same issue with the PIE results. The other bulk import that was affected is this one: Wildbook for Carnivores | Login
We did not delete that import; I asked the user to hold off doing anything with it until we knew what the problem & fix are. I’ll now get back to her and give her the option of deleting & re-uploading but she said the HotSpotter results were great so she might be fine with this batch as is or with the option of re-running matches from the encounters. I’ll see what she says.
Thanks for jumping on this one. I hope you had a great weekend.
Please have users proceed normally this week and let us know if the issue crops up again. As-is, we can’t reproduce the issue (whatever ML model state was sending zero scores has been overwritten), so we’ll want to see it happen and ideally have the ML team see it on a work day to review the state of WBIA to see what is happening.
We filed this as ticket SAGE-517. We’ve been seeing cases in PIE v2 models for other species (gray whales, whale sharks) recently wherein PIE v2 is heavily prefering matches to annotations with individual IDs and often presents no matches at all to annotations with no individual IDs. I don’t know if this is related to the hyena issue you saw or not, and weirdly, some of these results I included above actually challenge the nature of the issue as we can see hyena matches to un-ID’d annotations whereas for other species PIE v2 seems to be ignoring them. We’re digging into it, though it may be thorny and take time.
Please let us know if you see this issue crop up again as it may be related.