CI setup

[yarikoptic]

• running tests (see above) across all environments (separate environment per action), we also need separate workflow per producer (core, emph_testing_data, one per extension) ...

I think the course of action would be for now

• forget about extensions portion of the specification (yet to be figured out) and otherwise take current version in specification for the setup #5
• per each environment produce a github actions workflow (probably from a template) which would
  • to be triggered on cron (spread out in time, without times specified in spec, just based on index in the loop spreading an hour apart) or a PR (for now for any change here)
  • "docker" step: to ensure we have desired docker image here and on the docker hub (for future or local runs)
    • pull corresponding docker container. If no docker container yet present:
      • produce environments/Dockerfile.<environment>
      • build docker image
      • push to docker hub
  • install https://gin.g-node.org/dandi/nwb-healthstatus-samples into location known or provided to our helper.
  • "produce" step: to ensue we have desired data sample here and in the nwb-healthstatus-samples dataset (for future runs)
    • for each producer in data_producers if the environment matches:
      • run the create under that environment (create should just skip producing if file is already there)
    • datalad save updated state of the nwb-healthstatus-samples dataset
    • if any change was saved (HEAD points somewhere else that origin/master) -- datalad push it back
      • note: there could be race condition across parallel workflows run, so might need to do git pull --rebase or just git pull and try again? that should happen only for PR runs, and cron runs should be spread in time
  • "test" step: to actually run tests and report on their status
    • for each producer in data_tests if the environment matches:
      • run the test under that environment for that producer test cases
      • remember if any test run failed, so workflow "later" could announce to "fail" if any test fails

I have used producer term above but I guess it is more of a testsuite (core, spikeextractors, etc).

With the above plan ATM we can produce a very coarse dashboard where for each environment we would be able to say only if all good for it as a whole and not per producer. I think we will work out more detailed view later, possible following #10 idea.

• In principle we could produce workflow for environment x producer combination and thus readily/quickly provide a more complete dashboard... but not sure yet if worth going this way ATM.

[jwodder]

@yarikoptic

• Regarding the Docker step: Since images can't be pushed to Docker Hub without first creating a repository on the Hub, I think it'd be easier to just create the images at the same time that the repositories are created (either through building & pushing from the command line or by setting up automatic builds) and not bother with having the workflow build any images.
• Are you sure it's a good idea for PR runs to be pushing samples to the sample repo? It seems like pushes should only be done when running on master (e.g., as a scheduled run).

[yarikoptic]

• Regarding the Docker step: Since images can't be pushed to Docker Hub without first creating a repository on the Hub, I think it'd be easier to just create the images at the same time that the repositories are created (either through building & pushing from the command line or by setting up automatic builds) and not bother with having the workflow build any images.

I guess for now we could do it manually, but in the longer term I think it would be nice to have CI (not in PR but when ran in master) to take care about that. This way addition of a new environment would be as easy as editing a single file (the specification), sending a PR, seeing that it works, merging.

• Are you sure it's a good idea for PR runs to be pushing samples to the sample repo? It seems like pushes should only be done when running on master (e.g., as a scheduled run).

correct -- any pushes (docker or datalad) should happen only from master branch (scheduled) and indeed that should avoid race condition (unless cron runs overlap)

[jwodder]

@yarikoptic

• Please explain, step by step, what you want the first draft of the workflows to do.
• How are the workflows supposed to check out https://gin.g-node.org/dandi/nwb-healthstatus-samples such that it can be pushed to?

[yarikoptic]

• Please explain, step by step, what you want the first draft of the workflows to do.

Let's start with "docker" - "produce" steps in the issue description.
If any particular step is not clear - I would be happy to clarify, but need to know what is not clear.

• How are the workflows supposed to check out https://gin.g-node.org/dandi/nwb-healthstatus-samples such that it can be pushed to?

generate ssh keypair, add secret one into secrets, public to gin, share both with me via email