ACW bulk imports not processing? Update: no imports are going through IA

What Wildbook are you working in? ACW

What is the entire URL out of the browser, exactly where the error occurred?

Can you describe what the issue is you’re experiencing?
The owner of the bulk import in the link above reported that his batch seems to be stuck. The status shows that 2/24 detections have completed but these were added manually by the user so the detection algorithm doesn’t appear to be processing.

The bulk import spreadsheet has been sent to the services@ email address.

Thanks!
Maureen

Update: There are now another 2 uploads that are also not getting through detection, in addition to the one listed above:

Looks like a problem on the server side but I will send through the spreadsheets for these other uploads to services@ now.

thanks
Maureen

Thanks, @ACWadmin1

Thanks for sharing the spreadsheets. I don’t see issues in any of them; they look good to me.

Let me see what’s going on in the server side and I’ll update here.

2 Likes

We’ve restarted the server so detection jobs are now slowly but surely making their way through.

1 Like

Dear all,

I am the account holder who was experiencing an issue with my batches. It seems that the issue has been resolved as all my batches have been going through detection successfully. However, I am now experiencing an issue where the queue time increases when I send a batch for identification, and the process never finishes. At some point, I cannot see the estimated time, but instead, I receive an error message (please see attached screenshot).

Could you please assist me with resolving this issue? I appreciate your help in this matter.

Best regards,
Robin H.

Hi @rhorion - I’m glad we’ve got you through detection at least! How long have you been waiting for matching to complete on this batch? The “average number of minutes” that displays will change dynamically based on what’s currently being processed by the algorithm, which might not be your images yet; it could be someone else’s. And that number of minutes varies because each annotation has a set of criteria that it runs against in the database that cause the time it takes to complete matching to go up or down. For example, a wild dog in the location ID “Africa - South” will likely take MUCH longer than a hyena in Africa - West. And a left side wild dog in Africa - South will likely take longer than a wild dog with an “up” viewpoint because there will be far more of the former than the latter in the database. Although there are less match candidates for your batches based on the relatively low volume of data from Africa - West currently, you may still be behind a batch of wild dogs from Africa - South in the queue, if that makes sense. It’s not entirely a 1 batch at a time queue but somewhat in some cases.

The message of “attempting to fetch results” isn’t an indication of an error or a problem. The system simply hasn’t finished processing that particular annotation through the matching algorithm. The system will display the link to the match results page on the bulk import page almost immediately after you’ve sent it to ID but it doesn’t mean it’s finished searching for matches.

The system displays a few different messages like this that all indicate the same thing - it’s not done yet. The messages, if I remember correctly, include: “waiting for match results” and, hilariously but confusingly “gave up looking for match results”. All of these messages mean that the system is still working on finding match results.

If it doesn’t find any matches for that annotation, then it will display that annotation as the target image on the match results screen along with the message “did not find any matches”.

Only when the message on the match results screen actually uses the term “Error”, along with a long error code string, has there been a problem of some kind that you should report.

Matching can take quite a long time and is dependent on a number of factors: the number of images in the batch, the number of animals in each photo in the batch, the number of other bulk imports already processing through Image Analysis (IA), and the number of animals of the same species with the same or similar viewpoint, within the same location ID as your batch, in the database to be compared with.

Lastly, the matching algorithm runs from the first to the last encounter in your batch in order. They don’t always finish in the same order because a left side might take longer than a hind viewpoint, but you will start getting match results, generally, from the top of your bulk import list with the last one generally finishing last or close to last. So keep checking the status of your match results in that same order.

I usually recommend that users kick off Identification the night before they’re ready to start curating it to give it time to process. That said, when the system is really busy, as it has been the past few weeks (wet season!), it can take longer. Hopefully you won’t have to wait too much longer.

Sorry for the long reply but hopefully it was helpful!

best
Maureen

PS. Welcome to the Wildbook community!!!

Hi @rhorion!

Maureen is right; in this case, your match results aren’t appearing yet because they’re still processing. There are about 500 pending jobs in the ACW queue this morning so I’d recommend checking back later when it’s finished processing.

@ACWadmin1 Thanks again for your thorough breakdown of waiting for match results! After last week’s discussion in a similar thread I did end up adding more details to our FAQ page about detection and matching as well as making the page more visible in the table of contents.

1 Like

Thank you both for your quick answers.

I’ll wait and try to see if the identifications appear. However, for the first batch I’ve processed the identifications job was pretty quick (probably because we have very few or no data for West Africa) while it’s been 2 days for the new batches. And after few hours I am able to “re-send” the identifications (see screeshot attached).

Again, thank you,
Best,
Robin H.

Hi @rhorion - that “send to identification button” re-appears by design. It’s so that you can re-send the batch through ID again at any time. So not an issue and no effect on this batch being in the queue to get processed now. I’ll keep an eye on your batches as well but hopefully you’ll start seeing results soon.

best
Maureen

1 Like

Hi @Anastasia -

Unfortunately, I think there may be a problem on @rhorion’s ID jobs, after all:

These are all hyaena batches and in each case, HotSpotter has completed and is producing HotSpotter match results but PIE v2 is not displaying any results. I would have expected PIE to complete before HotSpotter but regardless, I’ve never seen match results like this where results are displayed for one of the algorithms but still showing “attempting to fetch results” for the other:
image

Also, the location ID for these batches is Africa - West and so there are very few images in the system for the algorithm to match against although that likely doesn’t matter from a ‘place-in-the-queue’ perspective but it should mean that once these batches get to the front of the queue, they process relatively quickly. 2 days does seem like a long time for any batch, but these in particular.

Is this normal? Or is it indicative of a problem?

thanks
Maureen

I see the same thing, too. Let me find out why ACW is being so difficult today.

1 Like

This was an interesting one to research. PIE is rejecting the black and white images due to the differing bit depth between those and color images. PIE doesn’t currently support 8-bit imagery. Anytime a black and white image comes up as a match in the import, it’s causing the job to fail, even when matching against color images.

The workaround to this is to convert the 8-bit greyscale images to 24-bit PNGs. You should be able to do this within MS Paint on Windows (i’m on a Mac otherwise I’d walk through the exact steps for how to do this).

You will also need to delete your bulk imports and re-upload them with the updated, 24-bit images. It’s important to get all traces of the 8-bit images out of the system in order to prevent ID breaking when attempting future matches.

I’m going to update our help docs to address this, as well.

Thank you for your very quick answer, as always!

It makes sense. I had a similar issue when running computer vision on this same dataset. I will do what you recommended when I have the time for it.

Do you think those steps could be directly integrated within ACW? The use of grey images in carnivore surveys is not ideal usually, but I think it still happens quite a lot in some projects. And if it needs a lot of pre-processing, it might be discouraging for some people.

edit : it seems that hotspotter is working (and pie for some images - very few). Is that ok for you if I don’t re-process and re-upload all images ? Hotspotter is enough for the IDs and I’ve already started.

Thank you a lot,
Robin H

I strongly recommend deleting and re-uploading the corrected images. The problem isn’t just that the 8-bit images can’t be read by PIE, it’s also that any time that 8-bit image comes up as a potential match against other animals, even in 24- or 96-bit color, those matches will also fail. So even if a second person uploaded images greater than 8-bits, when one of their images match against one of your 8-bit images, their Hotspotter matches will fail, too.

To clarify, this relates to Hotspotter detection across all Wildbooks that use it; not just ACW. Right now there’s no plan for this as it’s an uncommon issue (apparently it’s only come up once before). That could change if we see evidence to the contrary.

Ouch! That’s a nasty surprise for all of us but thanks for the additional information, @Anastasia. I’ll update all of my documentation as well and warn our new users.

@rhorion - I’m sorry for this additional inconvenience and I’m almost afraid to ask - is all the imagery you’ve collected from the 4 west African national parks in the same format? And is this 8-bit image format standard output from Pan’s camera traps?

This only appears to affect the black and white images from the upload, which appear to be night-time ones. During testing, the color images worked fine because they were 96-bit images. I’m not sure if it’s typical for trap cameras to alter the color depth for night and day photos, but if it’s possible to update the settings in the camera, it would save time later by avoiding manually converting the files.

We’ll definitely let new users know that. My concern is that, in this case, the entire conservation org that R works for has the same settings on the rest of their CTs. That would be a big change as it’s a large, international organization. But forewarned is fore-armed and thanks for the additional detail about it being specific to the night / B&W images only. Really appreciate all your help on this one!

1 Like

Thank you very much for the quick reply. I have deleted all my uploads containing grayscale images (which was almost all of them) and will try to upload them again (although the server seems to be down at the moment). I used a little Python code to look for and convert all 8-bit images to 24-bit images, and it seems to have worked. We will see if this solves the issue.

Regarding grayscale images, it is true that most researchers working on carnivores (especially for identification) will use flash or led at night. However, for some projects, particularly in areas with high poaching activity (such as West Africa), or when working with apes (to avoid disturbing the animals), the use of infrared cameras is preferable. These cameras take color images during the day and black and white images at night. In West Africa, we started with a mix of IR and flash cameras and are now switching to flash-only cameras. However, we still have some surveys with black and white images.

2 Likes

Thanks for doing that, @rhorion! That’s an interesting callout between IR and flash nighttime images and the low color depth issue. Since many ACW, Whiskerbook, and Wildbook for Lynx users submit their photos from camera traps and this is only the second time this issue has come up, I’m curious if IR photos are the primary issue or if it’s more about the camera’s bit-depth settings.