From ef74e4d0a6f4c63df51f1764e91fc4f24f9b6ef3 Mon Sep 17 00:00:00 2001 From: Denis Yuen Date: Thu, 16 Feb 2023 15:43:44 -0500 Subject: [PATCH 1/9] update readme --- README.md | 41 +++++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index 805d6df..bcbbc97 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ ![ga4gh logo](https://raw.githubusercontent.com/dockstore/dockstore-ui2/2.7.4/images/high-res/ga4gh.png) -[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3374001.svg)](https://doi.org/10.5281/zenodo.3374001) +[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.7079592.svg)](https://doi.org/10.5281/zenodo.7079592) Schemas for the GA4GH Tool Registry API @@ -13,15 +13,11 @@ This repository is the home for the schema for the GA4GH Tool Registry API. The The [Global Alliance for Genomics and Health](http://genomicsandhealth.org/) (GA4GH) is an international coalition, formed to enable the sharing of genomic and clinical data. -The GA4GH [Data Working Group](http://ga4gh.org/#/) concentrates on data representation, storage, -and analysis, including working with platform development partners and -industry leaders to develop standards that will facilitate -interoperability. - -Containers and Workflows Task Team +Cloud Work Stream ---------------------------------- -The Containers & Workflows working group is an informal, multi-vendor working group born out of the BOSC 2014 codefest, consisting of various organizations and individuals that have an interest in portability of data analysis workflows. Our goal is to create specifications that enable data scientists to describe analysis tools and workflows that are powerful, easy to use, portable, and support reproducibility for a variety of problem areas including data-intensive science like bioinformatics, physics, and astronomy; and business analytics such as log analysis, data mining, and ETL. +The Cloud Work Stream is focused on creating specific standards for defining, sharing, and executing portable workflows and accessing data across clouds. +We work with many different Driver Projects to develop, enhance, test, and use the Cloud Work Stream APIs. What is the Tool Registry API Schema? ------------------------------------- @@ -29,23 +25,25 @@ What is the Tool Registry API Schema? This is the home of the schema for the GA4GH Tool Registry API. The GA4GH Tool Registry API is a standard for listing and describing available tools (both stand-alone, Docker-based tools as well as workflows in CWL, WDL, Nextflow, Galaxy or Snakemake) in a given registry. This defines a minimal, common API describing tools that we propose for support by multiple tool/workflow registries like [Dockstore](https://www.dockstore.org/), [BioContainers](https://biocontainers.pro), and [Agora](https://github.com/broadinstitute/agora) for the purposes of exchange, indexing, and searching. This repo uses the [HubFlow](https://datasift.github.io/gitflow/) scheme which is closely based on [GitFlow](https://nvie.com/posts/a-successful-git-branching-model/). In practice, this means that the master branch contains the last production release of the schema whereas the develop branch contains the latest development changes which will end up in the next production release. -As of July 2019, this means that the 1.0.0 version is described on master whereas the develop branch contains the 2.0.0-beta.3 version which will evolve into the 2.0.0 production release. +As of February 2022, this means that the 2.0.1 version is described on master whereas the develop branch contains work which will accumulate and evolve into a 2.1 production release. -Our current proposal is to start with a read-only API due to potentially different views and approaches to registration/security. +Our current iteration focuses on a read-only API due to potentially different views and approaches to registration/security. -Key features of the current API proposal: +Key features of the current API: * read-only API -* May serve up CWL, WDL, Nextflow, Galaxy or Snakemake to describe a tool or represent a workflow - depending on the tool/workflow submitter -* ID: globally unique across systems and also identifies the system that it came from (ex: 123456323@agora.broadinstitute.org ) +* Serve up one or more of CWL, WDL, Nextflow, Galaxy or Snakemake to describe a tool or represent a workflow depending on the tool/workflow submitter +* Download individual workflow files or the entire workflow as a zip +* Interrogate the language versions of these workflows +* Get specific versions of workflows and tools, potentially with immutable versions with checksums on their files +* ID: globally unique across systems and also identifies the system that it came from (ex: 123456323@agora.broadinstitute.org ) Outstanding questions: * How do we track authorship? Should we track authorship of the tool metadata, the Docker image, or the underlying algorithm, or all of above? -* How to describe indexing and external services like an external [sparql](https://github.com/common-workflow-language/workflows#sparql) service. -* Terminology discussion (do we describe triples separately from tools? should we describe them as aggregations of tools for just the case that documents have more than one tool? etc.) +* How to describe indexing and external services like an external [sparql](https://github.com/common-workflow-language/workflows#sparql) service +* How to better co-ordinate with WES, TES How to view @@ -86,7 +84,10 @@ See the [LICENSE](LICENSE) For more information -------------------- -* http://genomicsandhealth.org/ -* [LICENSE](LICENSE) -* [Google Groups - old](https://groups.google.com/forum/#!forum/ga4gh-dwg-containers-workflows) -* [Google Groups - new](https://groups.google.com/a/genomicsandhealth.org/forum/#!forum/ga4gh-dwg-containers-workflows) +* [GA4GH Cloud Work Stream](https://github.com/ga4gh/wiki/wiki) - the wiki and meeting notes for the workstream +* APIs that we co-ordinate/meet with + * [DRS](https://github.com/ga4gh/wiki/wiki/Data-Repository-Service) + * [TES](https://github.com/ga4gh/wiki/wiki/Task-Execution-Service) + * [WES](https://github.com/ga4gh/wiki/wiki/Workflow-Execution-Service) +* [Global Alliance for Genomics and Health](https://www.ga4gh.org/) - GA4GH's main page +* [GA4GH Technical Alignment Sub Committee (TASC)](https://github.com/ga4gh/TASC) - we try to co-ordinate GA4GH API decisions here From e607bfa6c73747ccc8ef0ee9cb0a5bca2ffeb256 Mon Sep 17 00:00:00 2001 From: Denis Yuen Date: Thu, 16 Feb 2023 15:49:41 -0500 Subject: [PATCH 2/9] swagger and v2 clean-up https://github.com/ga4gh/tool-registry-service-schemas/issues/229 --- openapi/README.md | 2 +- openapi/openapi.yaml | 3 --- openapi/{ => v2}/ga4gh-tool-discovery.yaml | 0 3 files changed, 1 insertion(+), 4 deletions(-) rename openapi/{ => v2}/ga4gh-tool-discovery.yaml (100%) diff --git a/openapi/README.md b/openapi/README.md index 984b877..d835ea8 100644 --- a/openapi/README.md +++ b/openapi/README.md @@ -4,4 +4,4 @@ To make changes to the TRS, join the GA4GH organization or ask to join this repo - The openapi.yaml file with an OpenAPI 3 definition of the changes. - This openapi yaml file will be used in the TRS validation server. -The v1 directory should not be modified. It provides backward compatibility support for TRS servers/clients. +The v1 and v2 directories should not be modified. They provide a historical record and potentially backward compatibility support for TRS servers/clients. diff --git a/openapi/openapi.yaml b/openapi/openapi.yaml index c0a7220..e23aa8c 100644 --- a/openapi/openapi.yaml +++ b/openapi/openapi.yaml @@ -40,9 +40,6 @@ paths: application/json: schema: $ref: "#/components/schemas/Tool" - text/plain: - schema: - type: string "404": description: The tool can not be found. content: diff --git a/openapi/ga4gh-tool-discovery.yaml b/openapi/v2/ga4gh-tool-discovery.yaml similarity index 100% rename from openapi/ga4gh-tool-discovery.yaml rename to openapi/v2/ga4gh-tool-discovery.yaml From 2f526e6687fba9cca9b0b306396b445d98bada37 Mon Sep 17 00:00:00 2001 From: Denis Yuen Date: Thu, 16 Feb 2023 16:19:45 -0500 Subject: [PATCH 3/9] script seems broken with feature branches --- scripts/update-ghpages.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/update-ghpages.sh b/scripts/update-ghpages.sh index 4e98716..8330cb6 100644 --- a/scripts/update-ghpages.sh +++ b/scripts/update-ghpages.sh @@ -4,7 +4,7 @@ set -o pipefail set -o nounset set -o xtrace set -u -BRANCH=$(echo "${GITHUB_REF##*/}" | awk '{print tolower($0)}') +BRANCH=$(echo "${GITHUB_REF#refs/heads/}" | awk '{print tolower($0)}') BRANCH_PATH="preview/$BRANCH" mv preview preview2 git config --replace-all remote.origin.fetch +refs/heads/*:refs/remotes/origin/* From 4e14f63c5a46c973011e21fce6fcda3df351fedf Mon Sep 17 00:00:00 2001 From: Denis Yuen Date: Thu, 16 Feb 2023 17:17:08 -0500 Subject: [PATCH 4/9] Documentation updates for https://github.com/ga4gh/tool-registry-service-schemas/issues/114 --- README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index bcbbc97..120bca3 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ Schemas for the GA4GH Tool Registry API ======================================= -This repository is the home for the schema for the GA4GH Tool Registry API. The goal of the API is to provide a standardized way to describe the availability of tools and workflows. In this way, we can have multiple repositories that share Docker-based tools and WDL/CWL/Nextflow/Galaxy/Snakemake-based workflows and have a consistent way to interact, search, and retrieve information from these various registries. The end goal is to make it much easier to share scientific tools and workflows, enhancing our ability to make research reproducible, sharable, and transparent. +This repository is the home for the schema for the GA4GH Tool Registry API. The goal of the API is to provide a standardized way to describe the availability of tools and workflows. In this way, we can have multiple repositories that share containerized (or alternative containerized) tools and WDL/CWL/Nextflow/Galaxy/Snakemake-based workflows and have a consistent way to interact, search, and retrieve information from these various registries. The end goal is to make it much easier to share scientific tools and workflows, enhancing our ability to make research reproducible, sharable, and transparent. **See the human-readable [Reference Documentation](https://ga4gh.github.io/tool-registry-service-schemas). You can also explore the specification in the [Swagger Editor](https://editor.swagger.io/?url=https://raw.githubusercontent.com/ga4gh/tool-registry-schemas/develop/openapi/openapi.yaml).** *Manually load the JSON if working from a non-develop branch version.* Preview documentation from the [gh-openapi-docs](https://github.com/ga4gh/gh-openapi-docs) for the development branch [here](https://ga4gh.github.io/tool-registry-service-schemas/preview/develop/docs/index.html) @@ -22,7 +22,7 @@ We work with many different Driver Projects to develop, enhance, test, and use t What is the Tool Registry API Schema? ------------------------------------- -This is the home of the schema for the GA4GH Tool Registry API. The GA4GH Tool Registry API is a standard for listing and describing available tools (both stand-alone, Docker-based tools as well as workflows in CWL, WDL, Nextflow, Galaxy or Snakemake) in a given registry. This defines a minimal, common API describing tools that we propose for support by multiple tool/workflow registries like [Dockstore](https://www.dockstore.org/), [BioContainers](https://biocontainers.pro), and [Agora](https://github.com/broadinstitute/agora) for the purposes of exchange, indexing, and searching. +This is the home of the schema for the GA4GH Tool Registry API. The GA4GH Tool Registry API is a standard for listing and describing available tools (both stand-alone, containerized tools as well as workflows in CWL, WDL, Nextflow, Galaxy or Snakemake) in a given registry. This defines a minimal, common API describing tools that we propose for support by multiple tool/workflow registries like [Dockstore](https://www.dockstore.org/), [BioContainers](https://biocontainers.pro), and [Agora](https://github.com/broadinstitute/agora) for the purposes of exchange, indexing, and searching. This repo uses the [HubFlow](https://datasift.github.io/gitflow/) scheme which is closely based on [GitFlow](https://nvie.com/posts/a-successful-git-branching-model/). In practice, this means that the master branch contains the last production release of the schema whereas the develop branch contains the latest development changes which will end up in the next production release. As of February 2022, this means that the 2.0.1 version is described on master whereas the develop branch contains work which will accumulate and evolve into a 2.1 production release. @@ -37,7 +37,8 @@ Key features of the current API: * Interrogate the language versions of these workflows * Get specific versions of workflows and tools, potentially with immutable versions with checksums on their files * ID: globally unique across systems and also identifies the system that it came from (ex: 123456323@agora.broadinstitute.org ) - +* Refer to our [data model](https://ga4gh.github.io/tool-registry-service-schemas/DataModel/) for information on how to define TRS URIs if desired +* provides more structure than a simple un-formatted list of tools but it is also a standard for registries to implement as opposed to a registry implementation itself Outstanding questions: From 8ec056913b5144bea348fef5f005e703d9e8aa29 Mon Sep 17 00:00:00 2001 From: Denis Yuen Date: Thu, 16 Feb 2023 19:26:15 -0500 Subject: [PATCH 5/9] typo fix and trigger build --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 120bca3..a93d43a 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ How to contribute changes Take cues for now from the [CONTRIBUTING.md](https://github.com/ga4gh/tool-registry-service-schemas/blob/develop/CONTRIBUTING.md) document. -At the very least, create an issue in our [Github tracker](https://github.com/ga4gh/tool-registry-schemas/issues). +At the very least, create an issue in our [GitHub tracker](https://github.com/ga4gh/tool-registry-schemas/issues). Even better, fork the codebase, fix the issue, and create a pull request back to the project along with your ticket. From 6b175ec6f994c74f29fdf063baa1b2384bdf6853 Mon Sep 17 00:00:00 2001 From: Denis Yuen Date: Thu, 16 Feb 2023 19:29:06 -0500 Subject: [PATCH 6/9] fix link and really re-genrate TOC --- openapi/openapi.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openapi/openapi.yaml b/openapi/openapi.yaml index e23aa8c..3f6c6eb 100644 --- a/openapi/openapi.yaml +++ b/openapi/openapi.yaml @@ -19,7 +19,7 @@ tags: description: A group of web resources proposed as a common standard for tool repositories externalDocs: - url: https://ga4gh.github.io/tool-registry-service-schemas/Introduction/ + url: https://ga4gh.github.io/tool-registry-service-schemas/ paths: /service-info: $ref: https://raw.githubusercontent.com/ga4gh-discovery/ga4gh-service-info/v1.0.0/service-info.yaml#/paths/~1service-info From de1026992ed8550e2693d9defdcd5521f58f69cb Mon Sep 17 00:00:00 2001 From: Denis Yuen Date: Thu, 16 Feb 2023 19:33:41 -0500 Subject: [PATCH 7/9] prep for next version and even more re-generate --- openapi/openapi.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openapi/openapi.yaml b/openapi/openapi.yaml index 3f6c6eb..baf2ffe 100644 --- a/openapi/openapi.yaml +++ b/openapi/openapi.yaml @@ -13,7 +13,7 @@ info: described above. In practice, examples of "tools" include CWL CommandLineTools, CWL Workflows, WDL workflows, and Nextflow workflows that reference containers in formats such as Docker or Singularity. - version: 2.0.1 + version: 2.1.0 tags: - name: GA4GH description: A group of web resources proposed as a common standard for tool From 6b0600dd8683928b9dfb6e06f384e19152068000 Mon Sep 17 00:00:00 2001 From: Denis Yuen Date: Wed, 22 Feb 2023 15:13:49 -0500 Subject: [PATCH 8/9] propose tag-based search #205 --- openapi/openapi.yaml | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/openapi/openapi.yaml b/openapi/openapi.yaml index baf2ffe..38599c3 100644 --- a/openapi/openapi.yaml +++ b/openapi/openapi.yaml @@ -128,6 +128,15 @@ paths: description: Filter tools by the name of the descriptor type schema: $ref: '#/components/schemas/DescriptorType' + - name: tags + in: query + description: Filter tools by registry specific tags + schema: + type: array + items: + type: string + minItems: 1 + explode: false - name: registry in: query description: The image registry that contains the image. @@ -629,7 +638,7 @@ components: descriptor_type_version: type: object description: A map providing information about the language versions used in this tool. The keys should be the - same values used in the `descriptor_type` field, and the value should be an array of all the language versions used + same values used in the `descriptor_type` field, and the value should be an array of all the language versions used for the given `descriptor_type`. Depending on the `descriptor_type` (e.g. CWL) multiple version values may be used in a single tool. example: | @@ -726,7 +735,7 @@ components: - WDL - NFL - GALAXY - - SMK + - SMK DescriptorTypeVersion: type: string description: The language version for a given descriptor type. The version should correspond From 9760fee8b06c09cadab7984ef44081e87075065d Mon Sep 17 00:00:00 2001 From: Denis Yuen Date: Wed, 22 Feb 2023 15:16:11 -0500 Subject: [PATCH 9/9] Add tool version specific description #228 --- openapi/openapi.yaml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/openapi/openapi.yaml b/openapi/openapi.yaml index 38599c3..3cac981 100644 --- a/openapi/openapi.yaml +++ b/openapi/openapi.yaml @@ -656,6 +656,9 @@ components: description: Reports if this tool has a containerfile available. (For Docker-based tools, this would indicate the presence of a Dockerfile) + description: + type: string + description: The description of the tool version. This allows for documentation of a specific tool version as a tool evolves over time. meta_version: type: string description: The version of this tool version in the registry. Iterates when