Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom check in seprate package. #401

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

galachad
Copy link

@galachad galachad commented Nov 7, 2022

  • Add possibility build the DataQuality package on the top of DataQualityDashboard.

DataQualityDashboard

Why it's helpful:

  • Run custom checks on specific omop dataset.
  • Write custom sql checks.
  • Do not need to edit DataQualityDashboard when adding/removing checks.

Example useage:

DataQualityDashboard::executeDqChecks(connectionDetails = connectionDetails, 
                                    cdmDatabaseSchema = cdmDatabaseSchema, 
                                    resultsDatabaseSchema = resultsDatabaseSchema,
                                    cdmSourceName = "IBM_CCAE_cohort_123",
                                    cohortDefinitionId = 123,
                                    cohortDatabaseSchema = "IBM_CCAE.results"
                                    numThreads = numThreads,
                                    sqlOnly = sqlOnly, 
                                    outputFolder = outputFolder, 
                                    verboseMode = verboseMode,
                                    writeToTable = writeToTable,
                                    writeTableName = "dqdashboard_results_123",
                                    checkLevels = checkLevels,
                                    tablesToExclude = tablesToExclude,
                                    checkNames = checkNames,
                                    cdmVersion = "Custom5.4",
                                    systemFileNamespace = "PackageWithDefinedChecks")
                                    
# PackageWithDefinedChecks needs to contains the `Custom5.4` csv check files. 

* Add custom system.file package namespace

* SQL file source is taken from the package arg

* Update csv check description with packageName
@galachad
Copy link
Author

Hi @clairblacketer, do you have chance to take a look on this pr? Thanks

@katy-sadowski
Copy link
Collaborator

Hi @galachad, thanks very much for the submission! This is a nice & tidy solution to a challenge I'm sure many DQD users run into. We will discuss this in our next DQD team meeting and I'll get back to you about how we'd like to proceed.

In the meantime, just to confirm my understanding of the approach here:

  • SQL files for new custom checks are stored in the separate "PackageWithChecks", while the SQL files for the standard DQD checks stay in DataQualityDashboard
  • The packageName column in the Check Description files tells DQD where to look for the SQL file for a given check
  • Full copies of the check description & threshold files, containing descriptions of checks & thresholds for both the standard DQD checks and the custom checks, are stored in "PackageWithChecks"

@galachad
Copy link
Author

galachad commented Dec 5, 2022

Hi @katy-sadowski,
Yes, mainly that was the idea. I tested this approach in one of the project I'm working on.

Is it possible to join to DQD community and your meetings?

@katy-sadowski
Copy link
Collaborator

Is it possible to join to DQD community and your meetings?

Sure thing! Our next meeting is planned for next Thursday at 10am Eastern US time, and you can sign up to join here: https://forms.office.com/Pages/ResponsePage.aspx?id=lAAPoyCRq0q6TOVQkCOy1ZyG6Ud_r2tKuS0HcGnqiQZUOVJFUzBFWE1aSVlLN0ozR01MUVQ4T0RGNyQlQCN0PWcu

@katy-sadowski
Copy link
Collaborator

Hi again @galachad - @clairblacketer just added you to the DQD Working Group in Microsoft Teams. We'd love to have you join an upcoming meeting to present your PR and talk a bit about how you're using this feature at your institution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants