-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixing peek params for non-cgat-flow repositories #103
base: master
Are you sure you want to change the base?
Conversation
…ctive of value of working dir. This doesn't work if the pipeline is not a cgat-flow pipeline and pipeline_genesets is not in the same directory. Need to change operator priority of and and or
Hi, sorry not sure if im understanding your problem. Are you able to elaborate in more detail what you are specifically trying to do? Do you have test case? I am able to run pipelines outside the cgat-flow directory and don't have any issues. |
Most pipelines have a When you do These lines in if ("config" in sys.argv or "check" in sys.argv or "clone" in sys.argv and workingdir == "?!"):
workingdir = os.path.join(get_params()["pipelinedir"],
"pipeline_" + pipeline) The idea is that if There are two problems with this. The first is that ("config" in sys.argv) or
("check" in sys.argv) or
("clone" in sys.argv and workingdir == "?!") when we want it interpreted as ("config" in sys.argv or "check" in sys.argv or "clone" in sys.argv) and
(workingdir == "?!") so in the old formulation, This is a problem, because if you are developing a pipeline in a directory that does not contain To reproduce this behaviour, create the following new pipeline in a non cgatflow location, say from cgatcore import pipeline as P
from ruffus import originate
PARAMS = P.get_parameters(
[os.path.join(P.snip(__file__, ".py"), "pipeline.yml"),
"pipeline.yml"] )
PARAMS.update(P.peek_parameters(
PARAMS["annotations_dir"],
"genesets",
prefix="annotations_",
update_interface=True,
restrict_interface=True))
@originate("test")
def dummy(outfile):
P.touch(outfile) with a pipeline.yml in annotations:
dir: PATH_TO_ANNOTATIONS (with the actual path subbed in) - actually, it doesn't matter what you put here. Now go to a new directory and run $ python ~/devel/mypipe/pipeline_mypipe.py config Changing to the new formulation only solves half the problem though. Under the suggested change |
Thanks @IanSudbery I get it now. I will attempt to run your example |
Hi @IanSudbery , I managed to have a quick look at this. Could all of this not be handled in the PARAMS better by also parameterising the pipeline name? i.e.:
to
Then it wouldn't fallback to using pipeline_genesets? |
I don't understand? How is this different? Presumably you would put "genesets" in your template I wonder if the real solution is that if you are cloning, or contiging, or checking, you don't actually need to import the config values, so perhaps if |
Maybe I'm reading the code wrong, but the having annotation_dir: ?! still tries to load I agree that the solution should be return a warning (or at least a better warning that it returning that it can't find the working directory). |
Two things here:
at the moment is doesn't even check what is in `annotation_dir` most of the
time because the if statement is wrong.
But that aside, the point is that the pipeline name is not the name of
*this* pipeline, its the name of the external pipeline whose parameters one
wants to import. That extranal pipeline will always be the same for a given
script, irrespective of the fact particular that different runs of the
external pipeline might be required for different runs of the current
script.
Anyway, I think we have a solution? I would code it up, but I'm super busy.
I'll try to remember to do it before Christmas if not one else gets there
first.
|
To be honest I haven't looked at this in a lot of detail, iv had meningitis last week and still recovering, but I think the solution should work. |
Sorry to hear that Adam. Hope you are recovering well. I'll see what I can
do about implementing this.
|
Just wondering if this needed to be looked at again? |
This PR still fixes the problem of the incorrect grouping of conditions. The main fact that peek_parameters will still throw an error if run with a non-cgatflow pipeline remains. Its sat dormant because we've stopped really using peek_paramters as it just makes pipelines difficult to share as standalone entities. I suspect that your solution - return nothing, is still the correct one. |
peek_params was using pipeline directory when config was run, irrespective of value of working dir. This doesn't work if the pipeline is not a cgat-flow pipeline and pipeline_genesets is not in the same directory. Need to change operator priority of and and or.
Unfortunately this still doesn't fix the problem that it won't know where to look if workingdir is set to '?!', but its not in the cgat-flow directory.