In packages

Introduction

This vignette serves two distinct, but related, purposes:

Before we go on, we’ll attach the packages we use, expose the version of tidyr, and make a small dataset to use in examples.

library(tidyr)
library(dplyr, warn.conflicts = FALSE)
library(purrr)

packageVersion("tidyr")
#> [1] '1.0.2'

(mini_iris <- iris %>% 
    as_tibble() %>% 
    .[c(1, 2, 51, 52, 101, 102), ])
#> # A tibble: 6 x 5
#>   Sepal.Length Sepal.Width Petal.Length Petal.Width Species   
#>          <dbl>       <dbl>        <dbl>       <dbl> <fct>     
#> 1          5.1         3.5          1.4         0.2 setosa    
#> 2          4.9         3            1.4         0.2 setosa    
#> 3          7           3.2          4.7         1.4 versicolor
#> 4          6.4         3.2          4.5         1.5 versicolor
#> 5          6.3         3.3          6           2.5 virginica 
#> 6          5.8         2.7          5.1         1.9 virginica

Using tidyr in packages

Compared to dplyr and ggplot2, most tidyr functions have select semantics (like dplyr::select()), not the more common action semantics (like mutate(), filter(), arrange(), group_by(), …). This means that you typically provide tidyr functions with an expression that works with column names (like starts_with("x") or a:c) not with column values (like y = x * 2). As a consequence, it’s worthwhile to clarify the best patterns for calling tidyr functions inside another package.

There are three main cases that you’ll encounter:

Fixed column names

If you know the column names, this code works in the same way, in and out of a package:

mini_iris %>% nest(
  petal = c(Petal.Length, Petal.Width), 
  sepal = c(Sepal.Length, Sepal.Width)
)
#> # A tibble: 3 x 3
#>   Species    petal            sepal           
#>   <fct>      <list>           <list>          
#> 1 setosa     <tibble [2 × 2]> <tibble [2 × 2]>
#> 2 versicolor <tibble [2 × 2]> <tibble [2 × 2]>
#> 3 virginica  <tibble [2 × 2]> <tibble [2 × 2]>

But R CMD check will warn about undefined global variables (Petal.Length, Petal.Width, Sepal.Length, and Sepal.Width), because it doesn’t know that nest() is looking for the variables inside of mini_iris.

The easiest way to silence this note is to use one_of(). one_of() is a tidyselect helper (like starts_with(), ends_with(), etc.) that takes column names stored as strings:

mini_iris %>% nest(
  petal = one_of("Petal.Length", "Petal.Width"), 
  sepal = one_of("Sepal.Length", "Sepal.Width")
)
#> # A tibble: 3 x 3
#>   Species    petal            sepal           
#>   <fct>      <list>           <list>          
#> 1 setosa     <tibble [2 × 2]> <tibble [2 × 2]>
#> 2 versicolor <tibble [2 × 2]> <tibble [2 × 2]>
#> 3 virginica  <tibble [2 × 2]> <tibble [2 × 2]>

The tidyselect package offers an entire family of select helpers. You are probably already familiar with them from using dplyr::select().

Column names in a character vector

If the column names are in a character vector (possibly from a function call), you can provide that to one_of(), as above:

nest_egg <- function(data, cols) {
  nest(data, egg = one_of(cols))
}

nest_egg(mini_iris, c("Petal.Length", "Petal.Width", "Sepal.Length", "Sepal.Width"))
#> # A tibble: 3 x 2
#>   Species    egg             
#>   <fct>      <list>          
#> 1 setosa     <tibble [2 × 4]>
#> 2 versicolor <tibble [2 × 4]>
#> 3 virginica  <tibble [2 × 4]>

The use of one_of() here is important; if you don’t use it, and data contains a column named cols, nest() will nest it instead of the columns named in cols.

tidyselect interface

To provide an interface that works like the tidyr function that you’re wrapping, you should pass the argument along using {{ arg }}. {{ }} is a special tidy eval operator that captures the expression supplied by the user and forwards it to another tidy eval-enabled function.

nest_egg <- function(df, cols) {
  nest(df, egg = {{ cols }})
}

nest_egg(mini_iris, -Species)
#> # A tibble: 3 x 2
#>   Species    egg             
#>   <fct>      <list>          
#> 1 setosa     <tibble [2 × 4]>
#> 2 versicolor <tibble [2 × 4]>
#> 3 virginica  <tibble [2 × 4]>

For more complicated functions, you might want to use tidyselect directly:

sel_vars <- function(df, cols) {
  tidyselect::vars_select(names(df), {{ cols }})
}

sel_vars(mini_iris, -Species)
#>   Sepal.Length    Sepal.Width   Petal.Length    Petal.Width 
#> "Sepal.Length"  "Sepal.Width" "Petal.Length"  "Petal.Width"

(Many tidyr functions use ... so you can easily select many variables, e.g. fill(df, x, y, z). I now believe that the disadvantages of this approach outweigh the benefits, and that this interface would have been better as fill(df, c(x, y, z)). For new functions that select columns, please just use a single argument and not ....)

If you decide to support tidyselect syntax, I recommend re-exporting the select helpers so that users can use (e.g.) starts_with() without any extra work. There’s no helper to do this yet, but you can copy-and-paste the code that tidyr uses.

Travis-CI

Hopefully you’ve already adopted continuous integration for your package, in which R CMD check (which includes your own tests) is run on a regular basis, e.g. every time you push changes to your package’s source on GitHub or similar. The tidyverse team currently relies most heavily on Travis-CI for this, so that will be our example. usethis::use_travis() can help you get started.

We recommend adding a build to your matrix that targets the devel version of tidyr:

Example of .travis.yml config that tests against R devel, release, and oldrel, assesses test coverage, and includes a build against devel tidyr:

# R for travis: see documentation at https://docs.travis-ci.com/user/languages/r

language: R
cache: packages

matrix:
  include:
  - r: devel
  - r: release
    after_success:
    - Rscript -e 'covr::codecov()'
  - r: release
    name: tidyr-devel
    before_script: Rscript -e "remotes::install_github('tidyverse/tidyr')"
  - r: oldrel

tidyr v0.8.3 -> v1.0.0

v1.0.0 (aka v0.8.99.9000) makes considerable changes to the interface of nest() and unnest() in order to bring them in line with newer tidyverse conventions. I have tried to make the functions as backward compatible as possible and to give informative warning messages, but I could not cover 100% of use cases, so you may need to change your package code. This guide will help you do so with a minimum of pain.

Ideally, you’ll tweak your package so that it works with both tidyr 0.8.3 and tidyr 1.0.0. This makes life considerably easier because it means there’s no need to coordinate CRAN submissions - you can submit your package that works with both tidyr versions, before I submit tidyr to CRAN. This section describes our recommend practices for doing so, drawing from the general principles described in https://principles.tidyverse.org/changes-multivers.html.

If you use Travis-CI already, we strongly recommend adding a build that tests with the development version of tidyr; see above for details.

This section briefly describes how to run different code for different versions of tidyr, then goes through the major changes that might require workarounds:

If you’re struggling with a problem that’s not described here, please reach out via github or email so we can help out.

Conditional code

Sometimes you’ll be able to write code that works with v0.8.3 and v1.0.0. But this often requires code that’s not particularly natural for either version and you’d be better off to (temporarily) have separate code paths, each containing non-contrived code. You get to re-use your existing code in the “old” branch, which will eventually be phased out, and write clean, forward-looking code in the “new” branch.

The basic approach looks like this. First you define a function that returns TRUE for new versions of tidyr:

tidyr_new_interface <- function() {
  packageVersion("tidyr") > "0.8.99"
}

We highly recommend keeping this as a function because it provides an obvious place to jot any transition notes for your package, and it makes it easier to remove transitional code later on. Another benefit is that the tidyr version is determined at run time, not at build time, and will therefore detect your user’s current tidyr version.

Then in your functions, you use an if statement to call different code for different versions:

my_function_inside_a_package <- function(...)
  # my code here

  if (tidyr_new_interface()) {
    # Freshly written code for v1.0.0
    out <- tidyr::nest(df, data = one_of("x", "y", "z"))
  } else {
    # Existing code for v0.8.3
    out <- tidyr::nest(df, x, y, z)
  }

  # more code here
}

If your new code uses a function that only exists in tidyr 1.0.0, you will get a NOTE from R CMD check: this is one of the few notes that you can explain in your CRAN submission comments. Just mention that it’s for forward compatibility with tidyr 1.0.0, and CRAN will let your package through.

New syntax for nest()

What changed:

Why it changed:

Before and after examples:

# v0.8.3
mini_iris %>% 
  nest(Sepal.Length, Sepal.Width, Petal.Length, Petal.Width, .key = "my_data")

# v1.0.0
mini_iris %>% 
  nest(my_data = c(Sepal.Length, Sepal.Width, Petal.Length, Petal.Width))

# v1.0.0 avoiding R CMD check NOTE
mini_iris %>% 
  nest(my_data = one_of(c("Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width")))

# or equivalently:
mini_iris %>% 
  nest(my_data = -one_of("Species"))

If you need a quick and dirty fix without having to think, just call nest_legacy() instead of nest(). It’s the same as nest() in v0.8.3:

if (tidyr_new_interface()) {
  out <- tidyr::nest_legacy(df, x, y, z)
} else {
  out <- tidyr::nest(df, x, y, z)
}

New syntax for unnest()

What changed:

Why it changed:

Before and after:

nested <- mini_iris %>% 
  nest(my_data = c(Sepal.Length, Sepal.Width, Petal.Length, Petal.Width))

# v0.8.3 automatically unnests list-cols
nested %>% unnest()

# v1.0.0 must be told which columns to unnest
nested %>% unnest(one_of("my_data"))

If you need a quick and dirty fix without having to think, just call unnest_legacy() instead of unnest(). It’s the same as unnest() in v0.8.3:

if (tidyr_new_interface()) {
  out <- tidyr::unnest_legacy(df)
} else {
  out <- tidyr::unnest(df)
}

nest() preserves groups

What changed:

Why it changed:

If the fact that nest() now preserves groups is problematic downstream, you have a few choices:

Imagine we used group_by() then nest() on mini_iris, then we computed on the list-column outside the data frame.

(df <- mini_iris %>% 
   group_by(Species) %>% 
   nest())
#> # A tibble: 3 x 2
#> # Groups:   Species [3]
#>   Species    data            
#>   <fct>      <list>          
#> 1 setosa     <tibble [2 × 4]>
#> 2 versicolor <tibble [2 × 4]>
#> 3 virginica  <tibble [2 × 4]>
(external_variable <- map_int(df$data, nrow))
#> [1] 2 2 2

And now we try to add that back to the data post hoc:

df %>% 
  mutate(n_rows = external_variable)
#> Error: Column `n_rows` must be length 1 (the group size), not 3

This fails because df is grouped and mutate() is group-aware, so it’s hard to add a completely external variable. Other than pragmatically ungroup()ing, what can we do? One option is to work inside the data frame, i.e. bring the map() inside the mutate(), and design the problem away:

df %>% 
  mutate(n_rows = map_int(data, nrow))
#> # A tibble: 3 x 3
#> # Groups:   Species [3]
#>   Species    data             n_rows
#>   <fct>      <list>            <int>
#> 1 setosa     <tibble [2 × 4]>      2
#> 2 versicolor <tibble [2 × 4]>      2
#> 3 virginica  <tibble [2 × 4]>      2

If, somehow, the grouping seems appropriate AND working inside the data frame is not an option, tibble::add_column() is group-unaware. It lets you add external data to a grouped data frame.

df %>% 
  tibble::add_column(n_rows = external_variable)
#> # A tibble: 3 x 3
#> # Groups:   Species [3]
#>   Species    data             n_rows
#>   <fct>      <list>            <int>
#> 1 setosa     <tibble [2 × 4]>      2
#> 2 versicolor <tibble [2 × 4]>      2
#> 3 virginica  <tibble [2 × 4]>      2

nest_() and unnest_() are defunct

What changed:

Why it changed:

Before and after:

# v0.8.3
mini_iris %>% 
  nest_(
    key_col = "my_data",
    nest_cols = c("Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width")
  )

nested %>% unnest_(~ my_data)

# v1.0.0
mini_iris %>% 
  nest(my_data = one_of("Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"))

nested %>% unnest(one_of("my_data"))