[help] Nested static branching

[simonrolph]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Hi there, I have a question about the best practice for how to do static branching.

I want to use targets to support the data simulation process for method development/validation involving simulating an environment and then various data sampling procedures. This data generation process has a number of steps (currently 6 but might become more) and the branching can occur at each step. I might want to explore the impact of changes occurring at each data generation step. Each step is dependant on the output of the previous. I've put together a gist file with the full example: https://gist.github.com/simonrolph/735f4c6eb5994407535543225d7ad484

I made a dummy data generation process at the top of the script. I want to be able to set parameters for each of the data generation steps. So in this example I have a branching of 2, then each of these branches branch again into 2, then again into 3.

#here a user sets out what parameters they want to generate data using
#each parameter as a separate dataframe
par1 <- data.frame(par1_col = c(1, 2))
par2 <- data.frame(par2_col= c(10, 20))
par3 <- data.frame(par3_col = c(1, 2, 3))

#combined into one expanded grid dataframe
all_pars <- expand.grid(par1$par1,par2$par2,par3$par3)
names(all_pars) <- c("par1_col","par2_col","par3_col")

#the data generation processes
data_generation_process1 <- function(obj,par){
  obj*par
}
data_generation_process2 <- function(obj,par){
  obj/par
}
data_generation_process3 <- function(obj,par){
  obj+par
}

Just as a thought exercise I did it all in a single mapping. This overcomes the indentation step, but ignores that the branching happens at each of the data generation process so I'm doing lots of unneeded computation, so this is bad, although it's very readable:

list(
    tar_target(init_val,4),
    tar_map(values = all_pars,
            tar_target(step1,data_generation_process1(init_val,par1_col)),
            tar_target(step2,data_generation_process2(step1,par2_col)),
            tar_target(step3,data_generation_process3(step2,par3_col))
    )
  )

So I was first thinking I want to use 'static' branching as I am pre-defining the branches. First attempt using tar_map:

list(
    tar_target(init_val,4),

    tar_map(
      values = par1,
      tar_target(step1,data_generation_process1(init_val,par1_col)),

      tar_map(
        values = par2,
        tar_target(step2,data_generation_process2(step1,par2_col)),

        tar_map(
          values = par3,
          tar_target(step3,data_generation_process3(step2,par3))
        )
      )
    )
  )

This is fine, it's readable, however in my real workflow the increased indentation gets quite awkward. Also it's not possible to do tar_combine because I haven't defined my mappings (or so I believe)

I thought about re defining my mappings (in order to be able to do tar_combine) in advance but because of the nesting they end up in reverse order which is very counterintuitive, I want to write my pipeline starting at the beginning:

map3 <- tar_map(
    values = par3,
    tar_target(step3,data_generation_process3(step2,par3))
  )
map2 <- tar_map(
    values = par2,
    tar_target(step2,data_generation_process2(step1,par2_col)),
    map3)
map1 <- tar_map(
    values = par1,
    tar_target(step1,data_generation_process1(init_val,par1_col)),
    map2)
list(
    tar_target(init_val,4),
    map1
    )

I then looked at using dynamic branching and used tar_group_by. This is at least in the right order but it's not particularly readable as I need to create the a grouping target for each step.

list(
    tar_target(init_val,4),

    tar_group_by(grouped1,all_pars,par1_col),
    tar_target(step1,data_generation_process1(init_val, grouped1$par1_col),pattern = map(grouped1)),

    tar_group_by(grouped2,all_pars,par2_col),
    tar_target(step2,data_generation_process2(step1, grouped2$par2_col),pattern = map(grouped2)),

    tar_group_by(grouped3,all_pars,par3_col),
    tar_target(step3,data_generation_process3(step2, grouped3$par3_col),pattern = map(grouped3))
  )

In summary, all the solutions I came up don't seem quite satisfactory, primarily about the readability. Especially as I might want to share this pipeline and make it easy for someone to edit it and define their own data generation for their use-case.

I was wondering if anyone had any better ideas about how to implement nested static branching? Thanks

[wlandau]

I like pipeline 3 in the gist, but that's just my personal preference. For many of these options, it comes down to style. You can often make up for a lack of readability with the custom description/descriptions arguments of tar_target() and tar_map(), plus tar_network(label = "description")