Approval for array-based data structure and multi-tiered digests for sequence collections

[nsheff]

I would like to solicit feedback from community members on the latest iteration of the digest algorithm for sequence collection identifiers. After lots of discussion (see #8, #1), here is the latest proposal.

It's an array of arrays, and here I'm showing just 3 arrays, but this approach works for any number of arrays, and is backwards compatible with sequence collections that lack certain array definitions.

The retrieval works like this

A simple server could allow only recursion=1, but we agreed that recursion=0 is very useful and should probably be a required part of the specification, while recursion=2 should probably be disabled. Given that the =0 and =1 layers are possible, this also enables retrieval of components, which is independently valuable:

Regardless of what elements end up in a sequence collection, we're in a position to approve this as the digest algorithm. Feedback welcome!

[nsheff]

@jmarshall and @jb-adams would be interested to hear your thoughts.

[jb-adams]

I see this as a way to incorporate the two paradigms to sequence identifiers that we've been struggling to navigate:

1. vendor-neutral, checksum derived from only the sequence itself
2. vendor-specific, allows other fields (names, lengths) to be incorporated into the checksum

The result is a vendor-specific "meta-checksum" that contains within it the vendor-neutral checksum component. I think this is good, and a good way to get adopters to implement, but does raise a few questions:

• If the vendor-specific checksum is used primarily (passed to the service, referenced in papers, etc.), then does this lead to downstream difficulty in identifying equivalent collections across services, research papers, etc.? For example, if collection abcdef and collection 012345 had the same underlying sequences but different names, this fact would not be immediately apparent just by looking at the final checksum. This could be solved by the recursive lookup, but it's not easily apparent and requires API calls to investigate. This may not be a problem, but seems to be moving away from the transparency of the original refget spec.
• Thoughts around what checksum algorithms we support, and how we map across different identifiers for the same collection. If I provide an MD5 identifier do I only get MD5 identifiers for the components? Do we plan to support a mechanism by which I can get the ga4gh or trunc512 id of the components in the response? Again, would be difficult to see when 2 collections are equivalent if the ids in 2 published papers are using different algorithms. Also there might be bloating, performance issues if every component needs to support 3 ids. Simplest solution would be to have one algorithm for everything, but implementers may not like this.

My questions are mainly based on assessment of equivalency, and making sure we don't have a false-negative problem (2 collections are the same at the sequence level but appear different) when researchers share results. But if this is not a concern, then everything looks good to me.

[sveinugu]

I guess the following comment is relevant here, depending on whether the question of ordering is included in the definition of the "digest algorithm": #7 (comment)

[nsheff]

This was approved in the ADR with PR #14