specification for the setup

[yarikoptic]

environments:
  - name: _base_
    base-image: python
    apt:
      - libhdf5
  - name: pynwb-2.1.0
    pip:
      - pynwb==2.1.0
      - hdmf==1.1.0
  - name: pynwb-2.2.0
    pip:
      - pynwb==2.2.0
      - hdmf==1.1.0
  - name: spikeextractors-1.1.1:
      - pynwb=
      - spikeextractors=1.1.1
  - name: pynwb-latest
     pip: ["pynwb"] 
  - name: pynwb-dev
    run: pip install https://github.com/Neuro.../pynwb https://github.com/.../hdmf
data_producers:
    core:
      environments: [ 'pynwb-2.1.0', 'pynwb-2.3.0']
    emph_testing_data:
      environments: [ 'spikeextractors-1.1.1']
    extensions: # later might just sweep through all in https://nwb-extensions.github.io/
      _base-environments: ['pynwb-2.1.0']
      ndx-ibl-metadata:
         pip: ndx-ibl-metadata  # optional to map to pip name
         versions: [ '0.1.0', '1.0.0']
data_tests:
   core:
      environments: [ 'pynwb-2.1.0', 'pynwb-2.3.0', `pynwb-dev`]
     # optional: envs: to restrict to specific environments
     # maybe add urls to point to existing files in the archive, so tests could be ran against them

from the spec we

populate environments

environments/
   <environment.name>/Dockerfile, `environment_details.yaml`

which we would need to probably manually (or talk via docker hub API to establish the linkage from github actions?) to create docker containers.

produce sample files

to be uploaded (git annex copy or datalad push) to https://gin.g-node.org/dandi/nwb-healthstatus-samples/src/master/core under

<environment.name>/
   <data_producer>/
      <name, e.g. simple1>.nwb

or

<data_producer>/
    <name, e.g. simple1>[/_]<environment.name>.nwb

[yarikoptic]

might be relevant: https://github.com/ReproNim/testkraken/ as for "matricizing" the runs. I am not sure if it has separation needed for producer/test environment, and what conviniences would it provide us ATM (CC @satra) - or for this simple dashboard testkraken would not be appropriate since we are to use github actions and not some local deployment on HPC.

[satra]

let's ping @djarecka :)

[jwodder]

If I'm understanding testkraken correctly, the biggest hindrance is that we want the producers (corresponding to testkraken's "analysis") to produce multiple files, and we want the files to be tested in environments other than the ones they're produced in. I can't tell if testkraken can handle the first requirement, but it definitely doesn't seem to be made with the second requirement in mind.

Or is there a way to make such a setup work? @satra @djarecka

[djarecka]

Actually, the default is that the files are tested in one python environment.

[jwodder]

@djarecka Is there a way to test them in multiple environments? Also, is it possible to run just the "analysis" portion, save the produced files in a local directory, and run the "test" portion on the files at one's leisure?

[djarecka]

right now it's one testing environment and it does testing right after analysis, but if there is a need to do it, I could implement it