PIE V2 not producing useable results

Bug Reports
, ,
1

In which Wildbook did the issue occur?ACW

What operating system were you using? (eg. MacOS 10.15.3)WIN 10

What web browser were you using? (eg. Chrome 79)Chrome latest

What is your role on the site? (admin, researcher, etc)Admin

What happened? Feedback from 1 user and review of another user’s bulk import matching results shows PIE v2 producing garbage match results as well as errors on hyaena data.

What did you expect to happen?
Match results to be useful from PIE V2

What are some steps we could take to reproduce the issue?
Check bulk imports here

CC @ACWadmin1

@jason

2

All zero scores. Definitely problematic. Will raise with the ML team on Monday.

Thanks,
Jason

@parham @Drew

3

Thank you Jason, appreciate the quick response.

Paul

4

Hi @PaulK

Some good news here. This looks more like a hiccup than a PIE model failure. I am re-running your matches above and absolutely getting matches and correct ones (I think).

Here are example links I just ran:

Are all of these failures from the same bulk import? Have you done any curation on that data? One possibility is just to re-run these.

Thanks,
Jason

5

Thanks Jason.

I am going to cancel the bulk import and delete it. (Also going to get the User to break up his uploads into smaller batches!!)

Thanks for looking at this on a Saturday afternoon.

Regards

Paul

6

OK, please feel free to ping me if you see these again as they definitely were zero and wrong. I would like to see whether a bug caused them and if we can fix them.

Thanks,
Jason

8

Hi @ACWadmin1

Please have users proceed normally this week and let us know if the issue crops up again. As-is, we can’t reproduce the issue (whatever ML model state was sending zero scores has been overwritten), so we’ll want to see it happen and ideally have the ML team see it on a work day to review the state of WBIA to see what is happening.

Thanks,
Jason

9

Will do. Thanks again.

10

Hi @ACWadmin1

We filed this as ticket SAGE-517. We’ve been seeing cases in PIE v2 models for other species (gray whales, whale sharks) recently wherein PIE v2 is heavily prefering matches to annotations with individual IDs and often presents no matches at all to annotations with no individual IDs. I don’t know if this is related to the hyena issue you saw or not, and weirdly, some of these results I included above actually challenge the nature of the issue as we can see hyena matches to un-ID’d annotations whereas for other species PIE v2 seems to be ignoring them. We’re digging into it, though it may be thorny and take time.

Please let us know if you see this issue crop up again as it may be related.

Thanks,
Jason

1 Like