'grid' mode does not seem to work with Open Grid Scheduler

[samnooij]

I have been trying to get Jovian up and running on a cluster with Open Grid Scheduler, and now DRMAA is giving me this error when trying to submit the very first job:

drmaa.errors.DeniedByDrmException: code 17: ERROR! invalid option argument "-n"

It seems that it reads line drmaa: " -q bio -n {threads} -R \"span[hosts=1]\"" from profile/config.yaml and tries to submit jobs using that. However, this scheduler does not work with -n, but wants -pe [name of parallel environment] instead. (And I don't think it will work with -R\"span[hosts=1]\" either.) I tried updating the config file manually, but that did not work out.
Can I fix this to work with a different scheduling commands?

[DennisSchmitz]

Hmm, unfortunate that DRMAA doesn't interface with OGS. I do know there is a Snakemake grid config file (https://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html#cluster-configuration) but I have never tried it myself. But I think that is the way to go. Please let me know if I can help. If you want I can ask Laros et al. if they have experience with such a configuration?

[samnooij]

Yes, I think such a file will be necessary. Especially because this cluster will kill jobs if they take more than the reserved amount of RAM, and will soon be using a similar regime for runtime. I can try and make a draft configuration file.
However, having this file by itself does not seem to solve the issue that different schedulers require slightly different commands/parameters. So I still think there needs to be a way to adjust those to different environments.
For now I will ask the in-house Snakemake community for advice. See how other people got their pipelines working on the cluster.

P.S. My problem right now seems to be in the jovian script, lines 781-791 (specifically this line). There the parameters are hard-coded and overwritten in the profile/config.yaml file. Running snakemake by hand and using the --cluster parameter does get the pipeline running and jobs are submitted. (So far they all return with errors, but at least they are submitted.)

[samnooij]

I managed to get the pipeline running with a 'manual' snakemake command:
snakemake --profile profile --cluster "qsub -q all.q -pe BWA {threads} -l h_vmem={cluster.vmem} -cwd -j Y -V" -p --cluster-config profile/cluster_config.yaml

The config file that I used has already been uploaded to the repository. (Also see DennisSchmitz/jovian@1e1098f.) With this, it should be possible to run the pipeline on any Snakemake-compatible cluster system. However, since it does not use the bash jovian command, it may lack a few features that Jovian provides by default.

[DennisSchmitz]

Thanks for adding that Sam, I'm gonna reopen the issue for now as a reminder to add it to the documentation/FAQ.

Might also be worth updating the wrapper to function with a custom grid-configuration.

[samnooij]

I came across this article that describes another interesting approach of addressing different schedulers, Xenon: https://peerj.com/articles/8214/#p-13
I think we had heard of this already, but never really considered it. Maybe this can be put on the feature wishlist to facilitate the use of different HPC systems.