What is the entire URL out of the browser, exactly where the error occurred?
Can you describe what the issue is you’re experiencing? Matches are showing no results, but stuck on “attempting to fetch results”
Can you provide steps on how to reproduce what you’re experiencing? Send images to detection/identification, many images would return with this result after detection is completed.
If this is a bulk import report, send the spreadsheet to services@wildme.org with the email subject line matching your bug report
Status update from yesterday: the bulk import detection and ID that you kicked off processed without issue.
Clicking through the import’s encounters, I am seeing clean detections and ID results. This lends some evidence to the idea that we should only do detection and ID for one bulk import at a time.
Looking at your examples of missing results.
Example 1:
Failed result:
successful Re-run result via “Start another match”:
Example 2:
Failed result:
successful re-run result via “Start another match”:
Example 3:
actually did return a result
So overall, I am seeing matching working as designed.
With the missing matches, I believe we are seeing the result of the system being overloaded in the past with multiple, simultaneous bulk imports sent to detection and ID. As we discussed yesterday, only one bulk import should be run through detection and ID at a time because it also has implications for the question: what are we matching against? If multiple bulk imports are at interim stages going the the ML pipeline, the number of things to match against is constantly changing and we can’t be sure what we matched against because the ongoing processing is changing the number of things we match against constantly (e.g., adding new annotations every minute).
So we have two workarounds we can pursue:
If the number of missing results is low, these can simply be re-run manually using the menu option “start another match”
If the number of missing results is high, we can reset and re-run each bulk import in the order they were updated.
I’ve sent an email with this response already, and unless my other colleagues in the email chain disagree with it, or unless it causes more issues, I’d go with option 2.
This may be a fluke, as we ran previous bulk imports to detection and ID, one at a time, without a problem. Would it be possible to reset this bulk import as well? Or does something else need to be done?
I know for sure that I send both to identification, but now I’m not getting any results for either one. Should I just rerun identification for them again, one at a time?
We restarted your Image Analysis server, and it is back to processing these jobs. There are 450+ left to do. It will still take some time.
One thing to please keep in mind: as of today GrouperSpotter has requested 59,571 distinct image analysis jobs (detection and ID jobs)…in a matter of weeks. We have projects that are at the same number after being in operation for several years. We look forward to learning from this and growing our capacity to handle this volume of machine learning, but there will be growing pains along the way.
The system should be getting through these jobs over the next day. Progress on this job is now a little more visible via the link above.
Thanks for being so responsive and listening to/resolving our problems. It’s really cool seeing everything slowly pan out with the matches steadily coming in.