No matches returning, no result returning for images

Support
#1

What Wildbook are you working in? grouperspotter

What is the entire URL out of the browser, exactly where the error occurred?
https://www.grouperspotter.org/iaResults.jsp?taskId=dd45e340-2435-4a80-a7c8-9487b63b0693
https://www.grouperspotter.org/iaResults.jsp?taskId=1d29fca1-1af7-43aa-881d-826c0db7b0f8
https://www.grouperspotter.org/iaResults.jsp?taskId=ac346fc7-f60e-4b4b-aeb9-c5ec1b1fc4a9

Can you describe what the issue is you’re experiencing? Matches are showing no results, but stuck on “attempting to fetch results”

Can you provide steps on how to reproduce what you’re experiencing? Send images to detection/identification, many images would return with this result after detection is completed.

If this is a bulk import report, send the spreadsheet to services@wildme.org with the email subject line matching your bug report

#2

Hi @jason,

You mentioned during our meeting today that I should tag you to this post.

While this issue is being resolved is there anything we shouldn’t be doing in the meantime?

Thanks,
Bryant

1 Like
#3

Hi @BTran

Status update from yesterday: the bulk import detection and ID that you kicked off processed without issue.

https://www.grouperspotter.org/imports.jsp?taskId=3dc20ed3-77a0-4ca1-a9cf-e6cbe69e558f

Clicking through the import’s encounters, I am seeing clean detections and ID results. This lends some evidence to the idea that we should only do detection and ID for one bulk import at a time.

Looking at your examples of missing results.

Example 1:

Failed result:
https://www.grouperspotter.org/iaResults.jsp?taskId=dd45e340-2435-4a80-a7c8-9487b63b0693

successful Re-run result via “Start another match”:

https://www.grouperspotter.org/iaResults.jsp?taskId=7c1b1275-6975-4518-97e1-a6d6bc2073ff

Example 2:

Failed result:

https://www.grouperspotter.org/iaResults.jsp?taskId=1d29fca1-1af7-43aa-881d-826c0db7b0f8

successful re-run result via “Start another match”:

https://www.grouperspotter.org/iaResults.jsp?taskId=8f87cd5b-ed9c-4711-91fa-8df7f80ed186

Example 3:

https://www.grouperspotter.org/iaResults.jsp?taskId=ac346fc7-f60e-4b4b-aeb9-c5ec1b1fc4a9

actually did return a result

So overall, I am seeing matching working as designed.

With the missing matches, I believe we are seeing the result of the system being overloaded in the past with multiple, simultaneous bulk imports sent to detection and ID. As we discussed yesterday, only one bulk import should be run through detection and ID at a time because it also has implications for the question: what are we matching against? If multiple bulk imports are at interim stages going the the ML pipeline, the number of things to match against is constantly changing and we can’t be sure what we matched against because the ongoing processing is changing the number of things we match against constantly (e.g., adding new annotations every minute).

So we have two workarounds we can pursue:

  1. If the number of missing results is low, these can simply be re-run manually using the menu option “start another match”
  2. If the number of missing results is high, we can reset and re-run each bulk import in the order they were updated.

What would you prefer?

1 Like
#4

Hi @jason,

Thanks for getting back to me so quickly.

I’ve sent an email with this response already, and unless my other colleagues in the email chain disagree with it, or unless it causes more issues, I’d go with option 2.

-Bryant

#5

@jason

At this point in time, do we need to do anything on our end?

Thanks,
Bryant

#6

No, I will reset the bulk imports back to their pre-detection state today, and then you can re-run them one individually. I will update you shortly.

#7

Hi @BTran

The bulk imports have been reset, and you can now send them one at a time to detection and ID via the normal buttons.

Thanks,
Jason

1 Like
#8

Hi @jason,

Same issue is popping up in this bulk import: Grouper Spotter - Citizen science and facial recognition for grouper | Login

Some examples in the bulk import:
https://www.grouperspotter.org/iaResults.jsp?taskId=18f28353-3f9a-4f51-ad88-66114832f58a
https://www.grouperspotter.org/iaResults.jsp?taskId=abf9dba3-10ab-44d6-9a7b-f980b0755cf5

This may be a fluke, as we ran previous bulk imports to detection and ID, one at a time, without a problem. Would it be possible to reset this bulk import as well? Or does something else need to be done?

Thanks,
Bryant

#9

Hi Bryant,

I reset the import task:

https://www.grouperspotter.org/imports.jsp?taskId=1549ee30-afe4-4304-ac10-0e9705561faa

and it is ready for you to kick off detection and ID.

Please let me know if you still experience issues with this one.

Thanks,
Jason

#10

Hi @jason,

So there’s something new that came up.

I re-ran the bulk import (Grouper Spotter - Citizen science and facial recognition for grouper | Login) and then waited until the next day to view the results. Send the file to identification and everything, but no results, no bounding boxes nothing.

I believed there was no activity going on, so I moved to the next bulk import (Grouper Spotter - Citizen science and facial recognition for grouper | Login) to see if it was an issue with just that file. Waited another day, same results.

I know for sure that I send both to identification, but now I’m not getting any results for either one. Should I just rerun identification for them again, one at a time?

Thanks,
Bryant

#11

Hi @BTran

An update:

  • I updated your Wildbook to display additional stats about the bulk import, as well as to include links to the match results, so you can see how far along it is and look at results to date. You can see an example here:
    Grouper Spotter - Citizen science and facial recognition for grouper | Login

  • We restarted your Image Analysis server, and it is back to processing these jobs. There are 450+ left to do. It will still take some time.

One thing to please keep in mind: as of today GrouperSpotter has requested 59,571 distinct image analysis jobs (detection and ID jobs)…in a matter of weeks. We have projects that are at the same number after being in operation for several years. We look forward to learning from this and growing our capacity to handle this volume of machine learning, but there will be growing pains along the way.

The system should be getting through these jobs over the next day. Progress on this job is now a little more visible via the link above.

Thanks,
Jason

2 Likes
#12

About 97 matching jobs currently remain.

#13

Alright that sounds great,

Thanks for being so responsive and listening to/resolving our problems. It’s really cool seeing everything slowly pan out with the matches steadily coming in.

-Bryant

2 Likes
#14

One of the above bulk imports ran fine, but the other one (Grouper Spotter - Citizen science and facial recognition for grouper | Login) stopped halfway through? Not too sure what’s going on.

Thanks,
Bryant

#15

Or, is it actually completed, and that is as many matches as it could find within the folder?

#16

Hi @BTran

I have resent it and am monitoring today.

Thanks,
Jason

#17

Hi @BTran

It has completed.

Thanks,
Jason

1 Like
#18

A post was split to a new topic: Flukebook: long job wait time