Break data.table chain into two lines of code for readability

For many years, the way that automatic indentation in RStudio mis-aligns data.table pipes has been a source of frustration to me. I only recently realized that there is a neat way to get around this, simply by enclosing the piped operations in parentheses.

Here's a simple example:

x <- data.table(a = letters, b = LETTERS[1:5], c = rnorm(26))
y <- (
  x
  [, c := round(c, 2)]
  [sample(26)]
  [, d := paste(a,b)]
  [, .(d, foo = mean(c)), by = b]
  )

Why does this work? Because the un-closed parenthesis signals to the R interpreter that the current line is still not complete, and therefore the whole pipe is treated in the same way as a continuous line of code.

Chaining data.tables with magrittr

I have a method I'm using, with magrittr, using the . object with [:

library(magrittr)
library(data.table)

bar <- foo %>%
        .[etcetera] %>%
        .[etcetera] %>%
        .[etcetera]

working example:

out <- data.table(expand.grid(x = 1:10,y = 1:10))
out %>% 
  .[,z := x*y] %>% 
  .[,w := x*z] %>% 
  .[,v := w*z]
print(out)

Additional examples

Edit: it's also not just syntactic sugar, since it allows you to refer to the table from the previous step as ., which means that you can do a self join,

or you can use %T>% for some logging in-between steps (using futile.logger or the like):

out %>%
 .[etcetera] %>%
 .[etcetera] %T>% 
 .[loggingstep] %>%
 .[etcetera] %>%
 .[., on = SOMEVARS, allow.cartesian = TRUE]

EDIT:

This is much later, and I still use this regularly. But I have the following caveat:

magrittr adds overhead

I really like doing this at the top level of a script. It has a very clear and readable flow, and there are a number of neat tricks you can do with it.

But I've had to remove this before when optimizing if it's part of a function that's being called lots of times.

You're better off chaining data.tables the old fashioned way in that case.

You have to give a return between the [ and ] of each line. An example for how to divide your data.table code over several lines:

bar <- foo[, .(long_name_here = sum(foo2)), by = var
           ][order(-long_name_here)]

You can also give a return before / after each comma. An example with a return before the comma (my preference):

bar <- foo[, .(long_name_here = sum(foo2))
           , by = var
           ][order(-long_name_here)
             , long_name_2 := long_name_here * 10]

See this answer for an extended example