Snakemake remote rules re-read config file?

Question

I'm using snakemake (v7.22.0) to execute multiple jobs on a cluster. I have several different versions of the workflow which I somtimes execute in parallel, which involves changing the config file (and others). My problem is, if I understand the behavior correctly, that if I edit the config file before some of the jobs start running on the cluster, then they read the config file after I the edits, using different values from when the original workflow that created the job was created. In particular, this changes the workdir, so relative paths become incorrect.

I see this according to the job's stdout, which reiterates the config, e.g.

{'workdir': 'workdir1', 'seeds': ...}

but I get different values for 'workdir' for different jobs originating from the same snakemake session.

Is this indeed the behavior of snakemake in remote rules, or am I mistaken? If so, what I can do to be able to change the config and still get consistent results from all the remote jobs? Using multiple config files would lead to more human errors in my scenario.

Thanks in advance!

dariober · Accepted Answer · 2024-04-02 09:07:35Z

1

If I remember correctly and if things have not changed, snakemake re-executes the snakefile every time it submits a job to the cluster. So yes, if you change the config file half-way through the pipeline execution, the following jobs will use the updated config.

If different workflows use different configs, I would perhaps split the config file into a "common" file and a "workflow-specific" file. The two files will be then read by each version of the workflow.

Alternatively, have entries in your config file that instructs snakemake on what to use. E.g. your config file:

common_option1: foo
common_option2: bar
workflow_A:
    workdir: /some/path
workflow_B:
    workdir: /other/path

Then on top of your workflow file version "A":

WORKFLOW = 'workflow_A'

workdir = config[WORKFLOW]['workdir']

Then on top of your workflow file version "B":

WORKFLOW = 'workflow_B'

workdir = config[WORKFLOW]['workdir']

answered Apr 2, 2024 at 9:07

dariober

9,1024 gold badges32 silver badges54 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Yotam Feldman Over a year ago

Thanks for the answer! You wrote that it re-exectutes the Snakefile - so if the Snakefile changes, it could also be a problem? (And external scripts too, I guess?) Thanks again!

dariober Over a year ago

Yes, I think so, but maybe double check with a test workflow. See also snakemake.readthedocs.io/en/stable/project_info/…

dariober Over a year ago

By the way, regardless of snakemake's behaviour, editing the snakefiles to run different versions of the pipeline is not ideal practice since you easily loose track of what version produced a certain output. Better to run the same workflow with different configuration parameters.

Collectives™ on Stack Overflow

Snakemake remote rules re-read config file?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related