Persistent cloning issue in SeadragonSearch

[ChrissyTustison]

In which Wildbook did the issue occur?

SeadragonSearch

What operating system were you using? (eg. MacOS 10.15.3)

Mac OS 11.1 (not sure which systems submitters are using)

What web browser were you using? (eg. Chrome 79)

Chrome 90 (not sure which browsers submitters are using)

What is your role on the site? (admin, researcher, etc)

Admin

What happened?

We are continuing to see 30-40% of encounters duplicated after submission. It seems to happen regardless of location, photographer, photograph type, etc. Sometimes it’s one clone, and sometimes multiple clones. We’re concerned about the issue being so consistent and widespread, as well as the message it sends to our participants when we are regularly deleting their encounters. I think the takeaway is that we’d like to look into this more deeply and not allow it to become a normal part of the system.

Here is a handful of recent examples, and more can be provided if needed:

https://wildbook.dragonsearch.org/encounters/encounter.jsp?number=c4d16be9-fa33-4a09-9395-746e9af4b613

https://wildbook.dragonsearch.org/encounters/encounter.jsp?number=a5f4a388-ad65-495b-8fb2-20b14ea0cfb7

https://wildbook.dragonsearch.org/encounters/encounter.jsp?number=cfaf9ef4-ed58-48a6-89d3-1d2ddb72adaa

https://wildbook.dragonsearch.org/encounters/encounter.jsp?number=8ad3cf00-57ce-47a3-9ab9-cb3597b9631d

What did you expect to happen?

Didn’t expect cloned encounters to become such a consistent issue.

What are some steps we could take to reproduce the issue?

Could try submitting encounters and see if clones pop up.

If this is a bulk import report, send the spreadsheet to services@wildme.org with the email subject line matching your bug report

[jason]

Hi @ChrissyTustison,

The examples you show are definitely the system working as designed. In each of these cases, either two bodies or two heads were found, forcing a cloned Encounter to take one of the bodies/heads as the system believes two animals are present.

We may be at a point where we’re seeing the detector being too permissive and may need to adjust its operation. We’ll ask @parham about this.

[parham]

Hello everybody,

I’ve been doing some digging and, while I’m not able to identify an obvious error, it is the non-obvious errors we worry about as programmers. The best thing to do is to continue to document these kinds of errors and either we will suggest a slightly more strict configuration or we will see a pattern that we can identify and fix.

For now, it is a bit of a waiting game. We hope this is a relatively isolated incident and simply an artifact of the (sometimes) unpredictability of machine learning.

[ChrissyTustison]

Thanks @jason and @parham! Would it be helpful if I post a link in this thread every time an encounter is cloned? I didn’t want to inundate you with examples but I can if it would help to reveal the larger pattern

[parham]

This is a good place for it. We will just keep tabs on it.

[ChrissyTustison]

Hi all, just wanted to continue documenting examples here per our conversation the other day!

Here is a screenshot of an encounter that was cloned:

Here is a screenshot of the clone:

And here is the message shown in the metadata section:

Let me know if this is a helpful format for sharing and documenting these, or if there’s other info you’d like me to include or not include! Thanks, -C

[ChrissyTustison]

Also this is an example of the other kind of clone we were talking about where there only appears to be one annotation:

[ChrissyTustison]

Hi @jason , @parham , and @tanyastere !

Just wanted to follow up on this since our last meeting… Are the screen shots provided above helpful to you, and would you like us to continue documenting clones in this way, or adjust the way we are documenting? @NeridaWilson and I are happy to track this issue in whichever way provides the most useful information for you all, so just keep us posted on what you need. Thanks!

Cheers,
Chrissy

[parham]

Yes, this has been a helpful post to just aggregate all of the poor results. I haven’t seen a clear indicator yet as to what is going wrong, but this does provide us a good sense of frequency. In the last month or so, there seems to be a handful of poor sightings so far. Does this rate of failures seem under-reported or are there issues popping up more frequently?

[ChrissyTustison]

Thanks @parham ! There have actually been many more instances of cloning in the past month not reported here- we were waiting to see if this way of documenting was helpful (with the screen shots)… If it is, then I will begin sharing screen shots like the ones above every time it happens! Thanks, -C

[ChrissyTustison]

More examples coming through:

Original~

Clone~

[ChrissyTustison]

Original:

Clone:

[ChrissyTustison]

Original:

Clone:

[ChrissyTustison]

Original:

Clone:

[ChrissyTustison]

Original:

Clone:

[ChrissyTustison]

Original:

Clone:

[ChrissyTustison]

Hi @jason @parham @tanyastere,

This one is an example of what @NeridaWilson and I were describing at the meeting when we get a clone but can only see one annotation; not sure if this is the same issue as the others or something different:

https://wildbook.dragonsearch.org/encounters/encounter.jsp?number=c4d16be9-fa33-4a09-9395-746e9af4b613

[tanyastere]

Bookkeeping: tracking under SAGE-341