RF: move from single entrypoint API to entrypoint per command #309

jsheunis · 2023-06-02T21:10:56Z

The main goal of this PR is to close #245. IN the process, the PR evolved a lot and now includes these main changes:

update API to entrypoint per command: create, add, validate, serve, get, set, remove, translate, workflow
introducing datalad-next dependency, at first for constraint system but in future for whatever functionality is useful
refactor most of the code related to the main commands

Some specific outcomes to take note of:

Validation is now done according to the schema of a specific catalog (now located at path-to-catalog/schema/*), where previously validation was always done according to the latest schema of the installed package version (which is now the fallback).
Metadata can be passed to any catalog command that takes it as in argument, in different formats: json lines from STDIN, JSON-serialized string, and a file with JSON lines.
pytest fixtures are now located in tests/fixtures.py and exposed to all tests
catalog-get|set provides an extensible set of commands for configuring a catalog and reporting its properties
the handling of config files is refactored:
- The main changes relate to how/when a config file is provided.
- Previously, config files were specified during instantiation of the WebCatalog class, and a whole lot of obscure code was necessary to determine how this config applies to the catalog or dataset level.
- The refactored code receives the config file via the Create() and Add() commands or via the WebCatalog.create() or Node.add_attributes() python methods.
- Now, WebCatalog can be instantiated with location alone (no 'action' necessary anymore, and no config_file)
- Now, config is set on WebCatalog or Node instances after/during running their respective create() methods and not during instantiation
Translation functionality is refactored, and requires updated to existing translators:
- With the new catalog argument, translation occurs to the specific schema version of the catalog (if supported by an available translator); if the catalog argument is not supplied, the expected schema version is the latest supported by the package installation.
- The translator matching process is streamlined by keeping track of previously matched and instantiated translators
- Functionality to get the supported schema version, source name, and source version have been moved to the translator base class in order to support the abovementioned changes. This means existing and new translators will have to override these functions.

Issues that are addressed by this PR:

This commit starts the process of introducing multiple entrypoints by creating new and separate modules for 'create' and 'add'. It also adds the following changes: - introduces a dependency on datalad-next - introduces the use of dl-next's constraint system for parameter validation and coercion of new commands - allows multiple formats for the metadata argument to the 'add' command, including STDIN, stringified JSON, and a file with JSON lines. - introduces common fixtures via a new module and conftest - updates the tests associated with 'create' and 'add' - updates relevant test data - adds entrypoints for 'create' and 'add' commands The goal is for future commits adding more entrypoints to build on top of the basis established here.

This commit is necessary to allow validation according to the schema of a specific catalog, where previously validation was always done according to the latest schema of the installed package version. All catalog schemas now hava a dedicated location in 'catalog-name'/schema. If a metadata item is to be validated, this location will be inspected in order to build up a schema store that will be validated against. If this location does not exist, the fallback is the default schema store of the package installation. Subsequent changes include: - updating and moving packahe/catalog/tests/schema locations to the constants module - including the new schema location in package data - adding methods to the webcatalog class to locate the relevant schema store and validator - updating create and add commands and their tests accordingly

… to webcatalog instance method

…og-get' command

…on from a catalog, including metadata, config, and home page specs

This commit includes several changes that relate to the refactoring of the config functionality. The main changes relate to how/when a config file is provided. Previously, config files were specified during instantiation of the WebCatalog class, and a whole lot of obscure code was necessary to determine how this config applies to the catalog or dataset level. The refactored code receives the config file via the Create() and Add() commands or via the WebCatalog.create() or Node.add_attributes() python methods. Important outcomes of the changed code include: - WebCatalog can be instantiated with location alone (no 'action' necessary anymore, and no config_file) - config is set on WebCatalog or Node instances after/during running their respective create() methods and not during instantiation - WebCatalog class gained an 'add_record()' method to handle what was previously only done in Add() interface/command. - MetaItem instantiation now handles config_file input, in order to pass it to Node.add_attributes() at the appropriate time. Specific other changes include: - splitting Node class into its own module - updating utilities - updating tests to conform to the refactors code

…talog-set

catalog-translate command line entrypoint. Changes include: - using datalad-next constraints via validatedInterface class - introducing the catalog argument, which allows the translation to occur to the specific schema version (if supported by an available translator); if the catalog argument is not supplied, the expected schema version is the latest supported by the package installation. - streamlining the translator matching process by keeping track of previously matched and instantiated translators - moving functionality to get the supported schema version, source name, and source version to the translator base class. Existing and new translators will have to override these functions.

… to stdout

jsheunis · 2023-07-14T13:51:53Z

@mslw I think this comment from above is particularly relevant to you, since you also maintain a couple of translators:

Translation functionality is refactored, and requires updated to existing translators:

With the new catalog argument, translation occurs to the specific schema version of the catalog (if supported by an available translator); if the catalog argument is not supplied, the expected schema version is the latest supported by the package installation.

The translator matching process is streamlined by keeping track of previously matched and instantiated translators

Functionality to get the supported schema version, source name, and source version have been moved to the translator base class in order to support the abovementioned changes. This means existing and new translators will have to override these functions.

codecov-commenter · 2023-07-14T14:17:50Z

Codecov Report

Patch coverage: 91.45% and project coverage change: +3.81 🎉

Comparison is base (f612cb1) 82.32% compared to head (57bf6b4) 86.14%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #309      +/-   ##
==========================================
+ Coverage   82.32%   86.14%   +3.81%     
==========================================
  Files          32       43      +11     
  Lines        2173     2700     +527     
==========================================
+ Hits         1789     2326     +537     
+ Misses        384      374      -10

Flag	Coverage Δ
unittests	`79.00% <88.17%> (+7.24%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
datalad_catalog/__init__.py	`100.00% <ø> (ø)`
...lad_catalog/translators/bids_dataset_translator.py	`48.05% <ø> (ø)`
...lad_catalog/translators/datacite_gin_translator.py	`100.00% <ø> (ø)`
...lad_catalog/translators/metalad_core_translator.py	`91.66% <ø> (ø)`
...og/translators/metalad_studyminimeta_translator.py	`94.44% <ø> (ø)`
datalad_catalog/workflow.py	`67.70% <67.70%> (ø)`
datalad_catalog/catalog.py	`90.00% <75.00%> (+9.88%)`	⬆️
datalad_catalog/serve.py	`77.77% <77.77%> (ø)`
datalad_catalog/add.py	`80.95% <80.95%> (ø)`
datalad_catalog/webcatalog.py	`81.81% <81.31%> (-8.06%)`	⬇️
... and 26 more

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

I am committing this with a massive delay, but the changes seem to follow the refactoring of datalad-catalog in datalad/datalad-catalog#309 and specifically datalad/datalad-catalog@92ed3f7

jsheunis added 3 commits June 2, 2023 23:00

Adds the validate command, test, entrypoint, and a new constraint

2ea2221

jsheunis mentioned this pull request Jun 6, 2023

Consider catalog_add(notneeded) results #313

Closed

jsheunis and others added 11 commits June 8, 2023 13:50

minor tweaks to add and validate

b319a87

adds serve command and test; move serve implementation from interface…

cae9a78

… to webcatalog instance method

NF: add the get_record method to WebCatalog; to be used by new 'catal…

3848f19

…og-get' command

adds new ValidatedInterface, Get, which allows users to get informati…

bc9094e

…on from a catalog, including metadata, config, and home page specs

New tests and updated test data+fixtures for catalog-get

15a5dd3

Adds datalad-set entrypoint and tests

4a60328

Update catalog-get tests after implementing config refactoring and ca…

f7677f8

…talog-set

add the actual entrypoint for set

d345eb3

use dict.get to prevent keyerror

fceba64

Update datalad.tests.utils imports

292624a

jsheunis mentioned this pull request Jul 5, 2023

Define ideal user experience and workflow for catalog-translate #324

Closed

jsheunis added 14 commits July 7, 2023 09:31

doc tweaks

df15eea

update tests for new catalog-translate command

4ea3a35

update existing translators to accommodate translation refactoring

92ed3f7

code reformatting

e7681c5

stashed wip

5a66d8d

workflow refactoring wip

c951371

code reformatting

06092ee

update tests for catalog-workflow

6b2c10b

updates workflow test; remove outdated tests; update remaining tests

b48ec4b

Merge branch 'main' into all-of-the-entrypoints

3974983

use latest next

81985cf

Merge branch 'main' into all-of-the-entrypoints

d03840a

minimal fix for docs

7b9a7ee

jsheunis added 6 commits July 11, 2023 12:01

code formatting

41bc401

remove unnecessary imports

882bd8e

add remove cli to docs

96b9d17

report whether catalog record was added or updated

90737ad

use ui.message to output results of catalog-get and catalog-translate…

657e07b

… to stdout

code formatting

02f3ef3

let custom result renderer print json line for catalog-get

c6a8c80

let custom result renderer print json line for catalog-translate

57bf6b4

jsheunis force-pushed the all-of-the-entrypoints branch from 1ad9049 to 57bf6b4 Compare July 14, 2023 14:18

jsheunis added 2 commits July 18, 2023 23:33

updates docstrings and examples

807982a

Merge branch 'main' into all-of-the-entrypoints

50ce801

jsheunis force-pushed the all-of-the-entrypoints branch from abd029d to 50ce801 Compare July 18, 2023 21:36

update docs with information about new commands

dbd13cb

jsheunis merged commit ae17248 into main Jul 28, 2023
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RF: move from single entrypoint API to entrypoint per command #309

RF: move from single entrypoint API to entrypoint per command #309

jsheunis commented Jun 2, 2023 •

edited

Loading

jsheunis commented Jul 14, 2023

codecov-commenter commented Jul 14, 2023 •

edited

Loading

RF: move from single entrypoint API to entrypoint per command #309

RF: move from single entrypoint API to entrypoint per command #309

Conversation

jsheunis commented Jun 2, 2023 • edited Loading

jsheunis commented Jul 14, 2023

codecov-commenter commented Jul 14, 2023 • edited Loading

Codecov Report

jsheunis commented Jun 2, 2023 •

edited

Loading

codecov-commenter commented Jul 14, 2023 •

edited

Loading