ACW WBIA super slow - bulk imp. showing as queued since 4/12

Support
,
1

In which Wildbook did the issue occur? ACW

Hi, I’ve noticed a couple of weird things today that make me think that there might be a problem with WBIA.

1a) This bulk import was sent to identification 2-3 days ago and only 3 encounters completed ID; the rest are still showing as queued.
1b) Another bulk import from the same user uploaded on the same date is also showing as only partially completed.
1c) Bulk imports uploaded since the 12th (when the bulk imports above were uploaded & sent to IA) appear to be stuck in the queue as well:

  1. The ID average turnaround time seems massively high - 4498 minutes. Since we turned off hotspotter on all bulk imports, we haven’t seen anywhere near this kind of turnaround time - MiewID is crazy fast.

  2. During a training session this morning, I was getting 404 errors when I tried to click on inspect options in bulk imports that were processed in the past 5-7 days. I didn’t keep those tabs open so I can’t find an example right now.

So either the user who’s bulk imports in #1a & b above has a problem in one of his imports that’s caused the system to stall or there’s a different problem with WBIA.

I’ve emailed the user to ask for these 2 upload spreadsheets but he’s in Uganda so I won’t get a reply till tomorrow.

In the meantime, could someone have a look to see if WBIA is otherwise healthy?

thanks
Maureen

2

Thanks for the heads up! It looks like there was an issue with the AWS server. It should be back up and running now.

1 Like
3

Good morning @Anastasia.
Unfortunately, there’s still an issue somewhere. Some of the bulk imports that were stuck completed after yesterday’s fix but some of them have exception status on one or more encounters in each batch:

It appears that where PIE v2 is used, that algorithm completed all jobs (lions & hyenas). The exception status is on solely MiewID jobs.

Thanks for your help!
Maureen

4

I’ve resent the results with exceptions back through ID and it looks like they’re beginning to come back normally. Thanks for the heads up!

5

I hate to be the bearer of bad news but still having a problem here: Wildbook for Carnivores | Login

Even though I assume you already re-sent this one to ID, I just re-sent it again and it came back with exceptions. So sadly not fully resolved yet.

User report below of several bulk imports also showing exception status:

This one has quite a lot as I did manual annotations.

This one only has one exception but it’s because I did a manual annotation.

That the one you sent back for ID and that is currently also returning exceptions again.

6

Thanks for the update. I’ll let you know as soon as I have a status update on this.

1 Like
7

WBIA lost access to its GPU. We’ve restarted WBIA. Hopefully this will will get the exception jobs across the finish line. :crossed_fingers:

8

That seems to have fixed the problem. Thanks!

1 Like