Hotspotter so slow or not working

What Wildbook are you working in?

jaguar.wildbook.org

What is the entire URL out of the browser, exactly where the error occurred?
https://jaguar.wildbook.org/iaResults.jsp?taskId=94da1019-f145-44aa-ad49-d75a35c68a90

Can you describe what the issue is you’re experiencing?
I’m not sure why, but hotspotter is working unbelievably slow. After trying to match, then says waiting for results for a long time. Then the platform says: attempting to fetch results, with no turn bar and then an hour later still has not returned anything.

I thought the problem was not having any animals individual IDs finished, so I did individually ID about 6 or 7 animals using the visual matcher. Then, I tried to run hotspotter again and it still was doing the attempting to fetch results seemingly stuck.

When we did this during the intial tutorial with 3 images it was really fast, then I thought, well, it has to go through these 8-9k images now and is just taking a long time. I’m going to let it run overnight and see if it can complete one match.

Can you provide steps on how to reproduce what you’re experiencing?
Encounters>View Approved Encounters>choose some encounter from the list> select a photo and click the dropdown start match>click owner of data:my data click match

Hi Eve,

I see what you’re seeing. Several matching jobs across your data set took 10 seconds to 2 minutes on 10/12 when I was testing. Today however there are a batch of failed jobs and jobs taking 20+ minutes. Currently we’re back to 1-2 minutes per job. I’m having our development team take a look and filing support ticket WB-1039.

Thanks,
Jason

Hi Eve,

We restarted the jobs, and they completed normally and quickly. Matching jobs should generally take 2-3 minutes before results return. Occasionally matches up to 20 minutes may occur when new data needs to be rebuilt into the system cache, and these may back up other jobs in he queue. If you kick off many jobs in a row, they will back up in a queue and take longer, even with our newer multi-lane\multi-threaded queue system. But 2-3 minutes is the general benchmark.

Please let us know if you see this situation again.

Thanks,
Jason

Hi all, I last encountered the slow hotspotter last year in October as you can see from the previous thread. Now, I’ve uploaded new data and have been setting up 20 or so images to be run individually for hotspotter. Often it takes several hours and I might see three to 10 finish. Last night I let them run overnight. Today I setup a batch at 11am and most of them have still not finished. Some have been indefinitely attempting to fetch results. Normally if I wait a while and refresh the page it’s fine, although this time it seems to be stuck. Maybe it is high use on hotspotter and the server during the daytime.

Hi @evebohnett,

We’ve done over 120 Hotspotter jobs today on Whiskerbook. There are about 30 in the queue currently. The average job is taking 12 minutes, which is what is causing them to back up. We’ll take a look as to why they’re taking so long per job.

Thanks,
Jason

@evebohnett

All Hotspotter jobs have finished. We will migrate Hotspotter to a faster server early next week.

Thanks,
Jason

Okay great. Thanks so much. I hope moving hotspotter to a faster server is not only on account of me. I only have around 100 individuals left to match. which is probably 400 matches. I’m guessing I was the 120 hotspotter jobs. In any case, yes everything has run and it’s working!

1 Like

I have a pair of performance improvements I’m looking at across the Wildbooks that may also help move jobs through the queue as well.

Hi @evebohnett,

Just following up that we did upgrade Whiskerbook to a better server and added GPU support.

Thanks,
Jason