Match not found: GPU issue with MiewID

Hi Jason :slight_smile:

It looks like it happens again :

It also happens on other (but not all) pictures in the same encounter.

Maybe it is the new “no match found”, but i don’t think so ?

(Edit : Another example to help understanding what happens : Whiskerbook )

Hi @Lucas

OK, this is a separate issue. We have tracked this down to a GPU out-of-memory issue related to MiewID. Restarting the MiewID deep learning server solves the issue, restarts the jobs, and the links above are now completed with results.

We are working with our machine learning engineers to understand whether this is a memory leak or whether we simply need a more powerful GPU.

Thank you,
Jason

2 Likes

Thanks Jason :blush::+1:

Good luck with your investigation !!

housekeeping note: moving this to a separate thread now that we know this is a separate issue from originating post: "Attempting to fetch results" error in Whiskerbook - #3 by Lucas

1 Like

I created a ticket for this: Whiskerbook GPU out of memory · Issue #355 · WildMeOrg/Wildbook · GitHub

1 Like

This has been resolved. The Whiskerbook server was allowing up to three simultaneous MiewID/detector jobs (1 fast lane, two slow lane). We limited this to max 2 (1 slow, 1 fast) a week ago and have not observed any job problems of this nature since then.

1 Like

Amazing !!! :partying_face::clap::clap:

Thanks @Anastasia and your team :blush::+1:

1 Like