Slow matching from a bulk import again

What Wildbook are you working in?ACW

What is the entire URL out of the browser, exactly where the error occurred?

Can you describe what the issue is you’re experiencing?

Bulk import launched 3 days ago still not completed

FYI @ACWadmin1

@parham @MarkF

Can you provide steps on how to reproduce what you’re experiencing?

If this is a bulk import report, send the spreadsheet to services@wildme.org with the email subject line matching your bug report

Hi @PaulK

We’ve got a single job running really, really long, and it is holding up the Hotspotter queue.

Asking @parham for help examining this long match job.

Thanks,
Jason

1 Like

Hello all,

There were 72 jobs on the ACW background worker that were flagged as “suppressed”, which was the issue causing the delay. A job is suppressed when it fails to compute after 20 retries. There was a large amount of activity during this bulk upload, but also coincidentally large amounts of processing for other projects on the same machine. As a result, the jobs failed to complete and this has caused the bulk import to hang.

I have reset the compute server and reset and restarted these 72 jobs. It is currently processing them (successfully) at a rate of ~16 jobs per hour. We anticipate all of the jobs will be complete within the next 5 hours. I will be monitoring these jobs throughout the day and will respond to any weirdness or failed jobs, if I see them pop up again.

1 Like

Great

Thanks for the update and some visibility on how this stuff is managed on the backend.

Will let you know if we run into this again.

Regards

Paul

CC @ACWadmin1

1 Like

The ACW WBIA compute worker just finished catching up with the jobs that were submitted. Can you confirm if this bulk upload is appearing complete now?

@parham

I assume that the import you are referring to is this one?

When I examine it from the Sighting level, the 119 Media assets have resulted in 489 encounters.

Some of these are still giving me errors so I am inclined to say we may have some corrupt encounters.

Is it possible that we have run detector twice and hence are getting duplicate encounters?

Thanks

Paul

I’m honestly not entirely sure because the logs from the WBIA backend do not track work back to the specific encounters it corresponds to. I’m going to pull @MarkF in to help debug what is going on and how best to re-process or investigate this issue.

Thanks for the update.

Let me know if I can help at all.

Paul