Batch Import Issues

Support
,
1

What Wildbook are you working in?
Whiskerbook

What is the entire URL out of the browser, exactly where the error occurred?
Whiskerbook | Login?

I’m logged into the spoyafaryabi account.

Can you describe what the issue is you’re experiencing?
I had uploaded a set of data that we were planning to use to “verify how well the users are able to use Whiskerbook”. Originally we were going to use one set of data that I had previously matched in Whiskerbook and have another person to match the whole thing, and then changed our minds and have switched to some differet data.

Anyway, after I deleted the Batch Import from the Batch Import Log, the encounters are all still in the Approved Encounters list, which is one issue.

The other issue is that when I uploaded the new data just now, I selected the folder with only the 447 images of data, and when it got to the photo review verification page then it shows me 5500+ images that also included the previous batch (the one I had tried to delete).

Can you provide steps on how to reproduce what you’re experiencing?

Upload ~5000 photos, upload spreadhseet, decide you want to delete the whole thing, go to Batch Import log and delete the batch import. Look at the approved encounters and find they are still on the list but no photos are there.

Upload a new set of 477 photos, find out that the previous ~5000 photo data are still in the system, accept and move on, upload spreadsheet.

This may not be a big deal, although previously when I deleted a batch import everything deleted from the system including the approved encounters list. So I thought to ask if I had done something wrong in this case.

If this is a bulk import report, send the spreadsheet to services@wildme.org with the email subject line matching your bug report

2

Hello,

The visibility of the images from from the import during a subsequent import is normal. The reason is that many times a researcher will attempt an import and see that there are data entry errors and either stop before submitting the import, or remove a completed import with errors.

Because uploading the images can be a time consuming process we retain them from the first attempt. When you select a folder of images for a second attempt, the system checks if they are already saved and skips the ones that are, only uploading new ones. This can save a lot of time.

The second issue where the encounters remain after an import is removed should not occur. Can you please link me to the import, and to an example encounter? That will make the issue much easier to diagnose.

Thanks!

1 Like
3

Hi, @evebohnett !
I’m just following up on this. Were you able to resolve this? Sometimes, it just needs a good couple days to do its thing :crossed_fingers: .
If not, I’d like to, “yes, and” @colin and request a link to the entire bulk import page (if you still have access to it, which you may not because the bulk import logs get cleared from the user interface - but we can still pull them up from the database if needed, so no worries there).

Also, going forward, I would definitely recommend strongly against uploading 5,000 images in the same bulk import; that’s a BIG job for our detectors. We are currently recommending a maximum of 1000 at a time, and there is actually a ticket in progress to strictly enforce this in the near future (WB-1797).

Thanks,
Mark

4

Oh, great, thanks for getting back to me @colin and @MarkF Sorry I failed to respond earlier, I had a week of whirlwind that week as I recall, so this must have slipped through! Thanks for checking back in about this unresolved matter. Much appreciated.

Okay so, what I see is that the bulk import photo review page does still have the 5665 images total, including the 447 images I wanted to keep.
https://whiskerbook.org/import/reviewDirectory.jsp

I went into the batch import log and removed the ~5000 image upload after we decided not to use that dataset. I then uploaded 447 different images, and do see the import with 447 still in the log, which is fine.
https://whiskerbook.org/imports.jsp

However, the original ~300 encounters from the ~5000 photos still remain in the approved encounters. The images are gone but the encounters remain. Here is an example.
https://whiskerbook.org/encounters/encounter.jsp?number=729fcf6a-e0e3-481d-ae03-48b72977512d

It would be nice to have the system cleared of these encounters, although I understand if they cannot be. Anything with the spoyafaryabi username and the WakhanNP xxx sighting ID are the encounters I had wanted to delete. Perhaps I need to do this manually by removing the encounter from occurrence and then removing the occurrence?

Sorry about this. I have no idea what I did wrong or if there was a step I missed somewhere. I have done batch import removals before and simply uploaded a new spreadsheet to fix any data entry errors. This is the first time I did a bulk import deletion and then wanted to really delete the encounters and photos out of the system entirely.

Thanks for understanding. If anything is unclear please let me know. We do appreciate so much the support we have received for this project and platform.

1 Like
5

Hi, @evebohnett !

The images are gone but the encounters remain.

This is definitely weird. In my experience with bulk import deletion, it usually does a good job going through the dependency chain (e.g., marked individual → occurrence → encounter, etc.) and deleting things as needed to expunge those data from the database.

It sounds like there might be a bug with this functionality. I (or one of us) will check it out asap and get back to you.

Thank you for providing all of the helpful examples!
-Mark

6

I just deleted the 447 photos on spoyafaryabi’s account from my account evebohnett (because I can do that on my account as Jason made me an administrator recently).

This time all of those deleted and the encounters are all gone.

I have no idea what happened before. It may have been that I deleted spoyafaryabi’s images from evebohnett account previously but did not have the proper administrative permissions or something. It may have been that I was logged into spoyafaryabi’s account and did the bulk import deletion there.

If need be then I will go through and simply manually delete the encounters. I was thinking I could also just delete his account and start over with a new one as well because we have reorganized.

Sorry for all of these issue with the two accounts. We have been changing our plans/data as we change the methods, and now I think we will be all set.

Best,
Eve Bohnett

7

I thought to report that I recently was using the bulk import. I had previously imported a lot of images on several different accounts. I ran the detection algorithm on these data, and then deleted the bulk import. However, the encounters remained although the detections are gone.

I uploaded a new set of data and so now I have the encounters with the old deleted data, and the new data somehow.

Best,
Eve Bohnett