Add SVStratify and GroupedSVCluster tools #8990

mwalker174 · 2024-10-02T17:13:32Z

Implements two new tools and updates some methods for a revamp of the CombineBatches cross-batch integration module in gatk-sv.

SVStratify - tool for splitting out a VCF by variant class. Users pass in a configuration table (see tool documentation for an example) specifying one or more stratification groups classified by SVTYPE, SVLEN range, and reference context(s). The latter are specified as a set of interval lists using --context-name and --context-intervals arguments. All variants are matched with their respective group which is annotated in the STRAT INFO field. Optionally, the output can be split into multiple VCFs by group, which is a very useful functionality that currently can't be done efficiently with common commands/toolkits.
GroupedSVCluster - a hybrid tool combining functionality from SVStratify with SVCluster to perform intra-stratum clustering. This tool is critical for fine-tuned clustering of specific variants types within certain reference contexts. For example, small variants in simple repeats tend to have lower breakpoint accuracy and are typically "reclustered" during call set refinement with looser clustering criteria.
SVStratificationEngine - new class for performing stratification.
Updates to breakpoint refinement in CanonicalSVCollapser that should improve breakpoint accuracy, particularly in larger call sets. Raw evidence support and variant quality are now considered when choosing a representative breakpoint for a group of clustered SVs.
Added FlagFieldLogic type for customizing how BOTHSIDE_PASS and HIGH_SR_BACKGROUND INFO flags are collapsed during clustering.
RD_CN is now used as a backup if CN is not available when determining carrier status for sample overlap.
Removed no-sort option in favor of spooled sorting.
Bug fix: support for empty EVIDENCE info fields
Bug fix: in one of the JointGermlineCnvDefragmenter tests

This is a combination of 2 commits. Implement SVStratify Interchrom events Start unit tests Finish unit tests Multiple contexts per stratum Integration tests; fix empty context bug Test duplicate context name Handle CPX/CTX, add some tests Compiler error Prioritize SR and PE evidence types for representative breakpoint strategy Add SVStratifiedCluster Integration tests for SVStratifiedClustering Unused line Spooled sorting in cluster tools Start fixing JointGCNVSegmentation Fix JointGCNVSegmentation integration test Rename to GroupedSVCluster Fix sorting bug Comment Use RD_CN for CNV sample overlap; improve testing Documentation Add comment about requiresOverlapAndProximity Allow empty EVIDENCE list Add STRAT INFO field Representative breakpoint by variant quality Handle BOTHSIDES_SUPPORT and HIGH_SR_BACKGROUND in variant collapsing Add expected exception

ldgauthier

My only big ask is some clarification around "context" because it took me a long time to figure that out. Everything else you can take or leave.

ldgauthier · 2024-10-02T18:17:27Z

src/main/java/org/broadinstitute/hellbender/tools/sv/cluster/SVClusterLinkage.java

@@ -77,17 +77,17 @@ protected static boolean hasSampleOverlap(final SVCallRecord a, final SVCallReco
                final Set<String> samples = new HashSet<>(SVUtils.hashMapCapacity(genotypesA.size() + genotypesB.size()));
                samples.addAll(genotypesA.getSampleNames());
                samples.addAll(genotypesB.getSampleNames());
+                if (samples.isEmpty()) {
+                    // Empty case considered perfect overlap


For my own edification, why is that?

If it's a sites-only vcf then we don't need to enforce sample overlap.

ldgauthier · 2024-10-02T18:19:23Z

src/main/java/org/broadinstitute/hellbender/tools/sv/cluster/SVClusterLinkage.java

+            if (matchedSampleGenotype != null) {
+                return VariantContextGetters.getAttributeAsInt(matchedSampleGenotype, GATKSVVCFConstants.EXPECTED_COPY_NUMBER_FORMAT, 0);
+            } else {
+                return 0;


I understand we have to choose something as the default, but I feel weird defaulting to homozygous deletion. Why not -1?

Fair point. This code path shouldn't be possible so I changed it to throw an error. For the defaults, I made them -1 as a unique "null" value.

ldgauthier · 2024-10-02T18:22:21Z

src/main/java/org/broadinstitute/hellbender/tools/sv/cluster/SVClusterWalker.java

+    protected GATKPath outputFile;
+
+    /**
+     * Expected format is tab-delimited and contains a header with the first column SAMPLE and remaining columns


Do we have a tool that outputs this file? This is not familiar to me.

No this is a gatk-sv file designed to replace the ped file for storing allosome ploidies. We might make a gatk tool for this in the future though.

ldgauthier · 2024-10-02T18:23:21Z

src/main/java/org/broadinstitute/hellbender/tools/sv/cluster/SVClusterWalker.java

+     * clustering, but any associated annotation fields will be set to null in the output.
+     */
+    @Argument(
+            doc = "Fast mode. Drops hom-ref and no-call genotype fields and emits them as no-calls.",


I wouldn't call the attributes no-calls, I would call them missing

ldgauthier · 2024-10-02T18:24:42Z

src/main/java/org/broadinstitute/hellbender/tools/sv/cluster/SVClusterWalker.java

+     * Default genotypes are assigned when they cannot be inferred from the inputs, such as when VCFs with different
+     * variants and samples are provided.
+     */
+    @Argument(fullName = DEFAULT_NO_CALL_LONG_NAME,


By default you get 0/0 with no attributes?

Yes. Turns out this is sufficient for our batch clustering module, we just need the genotypes.

ldgauthier · 2024-10-08T17:15:10Z

src/test/java/org/broadinstitute/hellbender/tools/walkers/sv/SVStratifyIntegrationTest.java

+        runCommandLine(args, SVStratify.class.getSimpleName());
+    }
+
+    @Test(expectedExceptions = {GATKException.class, IllegalArgumentException.class})


I've been burned this way before where another class throws and IllegalArgumentException. I also made a note where the Utils arg validation is.

ldgauthier · 2024-10-08T17:17:42Z

src/test/java/org/broadinstitute/hellbender/tools/walkers/sv/SVStratifyIntegrationTest.java

+    public void testBwaMeltCohortRedundant() {
+        final File outputDir = createTempDir("stratify");
+        final String inputVcfPath = getToolTestDataDir() + "bwa_melt.chr22.vcf.gz";
+        final String configFile = getToolTestDataDir() + "test_config_redundant.tsv";


What happens if the config conflicts with the cmdline args?

It'll get caught in the Stratrum class and throw an error

ldgauthier · 2024-10-08T17:37:06Z

src/main/java/org/broadinstitute/hellbender/tools/sv/stratify/SVStatificationEngine.java

+     * @param svType SV type, may be null
+     * @param minSize minimum size in bp (inclusive), may be null
+     * @param maxSize maximum size in bp (exclusive), may be null
+     * @param contextNames reference context names


This confused me for a while because in the engine the ReferenceContext is like the actual FASTA sequence. You think this is obvious enough for users who haven't done GATK development?

Agreed, changed context to track

ldgauthier · 2024-10-08T17:38:42Z

src/test/java/org/broadinstitute/hellbender/tools/walkers/sv/SVStratifyIntegrationTest.java

+        expectedOutputSuffixes.put("DUP_lt5kb_RM", 0);
+        expectedOutputSuffixes.put("INV_gt1kb", 26);
+        expectedOutputSuffixes.put("BND_SD", 77);
+        expectedOutputSuffixes.put(SVStratify.DEFAULT_STRATUM, 1196);


I'm going to trust that you verified this with an orthogonal python script or somesuch

Yeah I've been testing against the gatk-sv pipeline results, although those evaluations are qualitative. I trust this will alert us if any unintentional changes happen though.

ldgauthier · 2024-10-08T17:42:46Z

...g/broadinstitute/hellbender/tools/sv/stratify/SVStratificationEngineArgumentsCollection.java

+    public GATKPath configFile;
+
+    @Argument(
+            doc = "Context intervals file. Can be specified multiple times.",


I would call this something more like "repeat track intervals file". I'm sure you could generalize beyond repeats, but I found this really vague.

now "Track intervals file" if that works?

mwalker174 requested a review from ldgauthier October 2, 2024 17:13

Add vcf splitting as an option

473d484

mwalker174 mentioned this pull request Oct 3, 2024

Update CombineBatches workflow broadinstitute/gatk-sv#732

Draft

ldgauthier approved these changes Oct 8, 2024

View reviewed changes

Reviewer comments

8751795

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SVStratify and GroupedSVCluster tools #8990

Add SVStratify and GroupedSVCluster tools #8990

mwalker174 commented Oct 2, 2024

ldgauthier left a comment

ldgauthier Oct 2, 2024

mwalker174 Oct 24, 2024

ldgauthier Oct 2, 2024

mwalker174 Oct 24, 2024

ldgauthier Oct 2, 2024

mwalker174 Oct 24, 2024

ldgauthier Oct 2, 2024

ldgauthier Oct 2, 2024

mwalker174 Oct 24, 2024

ldgauthier Oct 8, 2024

mwalker174 Oct 24, 2024

ldgauthier Oct 8, 2024

mwalker174 Oct 24, 2024

ldgauthier Oct 8, 2024

mwalker174 Oct 24, 2024

ldgauthier Oct 8, 2024

mwalker174 Oct 24, 2024

ldgauthier Oct 8, 2024

mwalker174 Oct 24, 2024

Add SVStratify and GroupedSVCluster tools #8990

Are you sure you want to change the base?

Add SVStratify and GroupedSVCluster tools #8990

Conversation

mwalker174 commented Oct 2, 2024

ldgauthier left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment