Not sure when idenfication for bulk import will be completed

What Wildbook are you working in? GrouperSpotter

What is the entire URL out of the browser, exactly where the error occurred?
The bulk import:
https://www.grouperspotter.org/imports.jsp?taskId=ff719d44-7b4a-477e-a29f-332c441b3c4a

Example match results/encounters in bulk import:
https://www.grouperspotter.org/iaResults.jsp?taskId=96daba40-a7a2-4d3b-b313-6ffa478681b5&scoreType=image&nResults=2

https://www.grouperspotter.org/encounters/encounter.jsp?number=975e143d-707b-4bd2-8991-87c5388bc65d

https://www.grouperspotter.org/iaResults.jsp?taskId=e5e3dbf4-1082-4031-bfb7-d571f551082a

Can you describe what the issue is you’re experiencing? Some match results display nothing, others display that they are fetching results, while some encounters have no match results.

I believe this may be in part because of the size of my bulk import, but it has been a few days at this point and I’m wondering if these are the results after the system has completed identification, or if the system is still working through this bulk import? How will I know for sure if identification is completed?

Can you provide steps on how to reproduce what you’re experiencing? Bulk import and examples are linked above. I’m attempting to look at match results for encounters in bulk import.

If this is a bulk import report, send the spreadsheet to services@wildme.org with the email subject line matching your bug report

Hi, @BTran!
I find myself wondering whether this has to do with the half-deleted bulk import. Maybe the system barfed a little bit when it found assets that it was trying to identify that were missing because of deletion.

I’ll plan on looking into this in more detail on Thursday/Friday of this week.
Thanks,
Mark

Hi @MarkF,

Thank you for your reply.

I want to also say that if it’s possible, we’d be okay having everything removed and starting off with a clean slate on the website, if it makes things easier.

We have the spreadsheets and all the photos saved up for bulk importing, so we can reupload everything again in smaller batches if it comes down to that.

Thanks,
-Bryant

Hi again @MarkF,

Sorry for this barrage of messages. I understand you’re busy but:

Have there been any updates to this?

Thanks again,
Bryant

Hi @BTran !
We’ve got quite a backlog.

I think a workaround for you in the meantime would be just doing a bunch of smaller bulk imports. The duplicated encounters from the current bulk import will have different uuids, so deleting them later won’t affect ones that you upload now.

-Mark

Hi again @MarkF

Another issue with some of the smaller bulk imports I wanted to bring up.

The first few bulk imports seemed to work alright, but then a few of the more recent ones would confirm that images were sent to identification but then would not actually do anything when checked on after a few hours (ends up asking if we want to send to identification again):
https://www.grouperspotter.org/imports.jsp?taskId=b8a72f09-975f-4290-8ee6-06a1b2542ce7

Before and after images of the same bulk import (linked above):

thanks,
Bryant

Hi, @BTran !
Our machine learning expert reported some slow behavior on grouper over the weekend and into today. I’ll look at this asap. There’s been a lot of activity on community lately, so the backlog has been growing. But I have not forgotten you, and you’re definitely on the queue!
-Mark

Hey @MarkF,

That’s great to hear about the queue.

Would you be comfortable of giving an ETA when you’ll be able to look into these issues on grouperspotter? Later next week, within the month, etc…?

-Bryant

Hi, @BTran !
We are noticing similar issues in a few other wildbooks.
I can’t give you a solid timeline because we don’t yet fully understand the nature of the problem. I certainly think that, “within the month” is realistic for a full fix, but we can definitely try to work around it!
Are you able to delete the new, smaller bulk imports and retry those that failed?
-Mark

Hi, @BTran !
We killed the jobs on grouperspotter and re-started them.
It’s got ~41,000 jobs in its queue (the samples get compared against each other).

Can I request that you don’t submit any new bulk imports in next, say 4-5 days?

In the future, if y’all anticipate having huge bulk imports like this, limiting the ID by locationID might be super useful for speeding ID up.

-Mark

Hey @MarkF,

Thanks for the quick response on the other post.

I’m currently deleting the large bulk import right now, but afterwards, can we continue uploading smaller bulk imports to the site, at this point in time?

-Bryant

Also, I’ve been reading through some of the documentation on image analysis and the matching system used by WildMe, but I’m not too familiar in how everything works just yet.

Are matches limited to the images provided within the bulk import? Do they look at previous bulk imports as well? If we wanted to compare all images to all other images, should we upload everything first and then start the matching process for each bulk import?