Tidier.jl is a data analysis package inspired by R's tidyverse and crafted specifically for Julia. Tidier.jl is a meta-package in that its functionality comes from a series of smaller packages. Installing and using Tidier.jl brings the combined functionality of each of these packages to your fingertips.
There are 2 ways to install Tidier.jl: using the package console, or using Julia code when you're using the Julia console. You might also see the console referred to as the "REPL," which stands for Read-Evaluate-Print Loop. The REPL is where you can interactively run code and view the output.
Julia's REPL is particularly cool because it provides a built-in package REPL and shell REPL, which allow you to take actions on managing packages (in the case of the package REPL) or run shell commands (in the shell REPL) without ever leaving the Julia REPL.
To install the stable version of Tidier.jl, you can type the following into the Julia REPL:
]add Tidier
The ] character starts the Julia package manager. The add Tidier command tells the package manager to install the Tidier package from the Julia registry. You can exit the package REPL by pressing the backspace key to return to the Julia prompt.
If you already have the Tidier package installed, the add Tidier command will not update the package. Instead, you can update the package using the the update Tidier (or up Tidier for short) commnds. As with the add Tidier command, make sure you are in the package REPL before you run these package manager commands.
If you need to (or prefer to) install packages using Julia code, you can achieve the same outcome using the following code to install Tidier:
import Pkg
Pkg.add("Tidier")You can update Tidier.jl using the Pkg.update() function, as follows:
import Pkg; Pkg.update("Tidier")Note that while Julia allows you to separate statements by using multiple lines of code, you can also use a semi-colon (;) to separate multiple statements. This is convenient for short snippets of code. There's another practical reason to use semi-colons in coding, which is to silence the output of a function call. We will come back to this in the "Getting Started" section below.
In general, installing the latest version of the package from the Julia registry should be sufficient because we follow a continuous-release cycle. After every update to the code, we update the version based on the magnitude of the change and then release the latest version to the registry. That's why it's so important to know how to update the package!
However, if for some reason you do want to install the package directly from GitHub, you can get the newest version using either the package REPL...
]add Tidier#main
...or using Julia code.
import Pkg; Pkg.add(url="https://github.com/TidierOrg/Tidier.jl")Once you've installed Tidier.jl, you can load it by typing:
using TidierWhen you type this command, multiple things happen behind the scenes. First, the following packages are loaded and re-exported, which is to say that all of the exported macros and functions from these packages become available:
- TidierData
- TidierPlots
- TidierCats
- TidierDates
- TidierStrings
- TidierText
- TidierVest
Don't worry if you don't know what each of these packages does yet. We will cover them in package-specific documentation pages, which can be accessed below. For now, all you need to know is that these smaller packages are actually the ones doing all the work when you use Tidier.
There are also a few other packages whose exported functions also become available. We will discuss these in the individual package documentation, but the most important ones for you to know about are:
- The
DataFrame()function from the DataFrames package is re-exported so that you can create a data frame without loading the DataFrames package. - The
@chain()macro from the Chain package is re-exported, so you chain together functions and macros - The entire Statistics package is re-exported so you can access summary statistics like
mean()andmedian() - The CategoricalArrays package is re-exported so you can access the
categorical()function to define categorical variables - The Dates package is re-exported to enable support for variables containing dates
Before we dive into an introduction of Julia and a look into how Tidier.jl works, it's useful to show you what Tidier.jl can do. First, we will read in some data, and then we will use Tidier.jl to chain together some data analysis operations.
This dataset comes with the Ecdat R package and and is titled OFP. You can read more about the dataset here. To read in datasets packaged with commonly used R packages, we can use the RDatasets Julia package.
julia> using Tidier, RDatasets
julia> ofp = dataset("Ecdat", "OFP")
4406×19 DataFrame
Row │ OFP OFNP OPP OPNP EMR Hosp NumChro ⋯
│ Int32 Int32 Int32 Int32 Int32 Int32 Int32 ⋯
──────┼────────────────────────────────────────────────────
1 │ 5 0 0 0 0 1 ⋯
2 │ 1 0 2 0 2 0
3 │ 13 0 0 0 3 3
4 │ 16 0 5 0 1 1
5 │ 3 0 0 0 0 0 ⋯
6 │ 17 0 0 0 0 0
7 │ 9 0 0 0 0 0
⋮ │ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋱
4401 │ 12 4 1 0 0 0
4402 │ 11 0 0 0 0 0 ⋯
4403 │ 12 0 0 0 0 0
4404 │ 10 0 20 0 1 1
4405 │ 16 1 0 0 0 0
4406 │ 0 0 0 0 0 0 ⋯
13 columns and 4393 rows omittedNote that a preview of the data frame is automatically printed to the console. The reason this happens is that when you run this code line by line, the output of each line is printed to the console. This is convenient because it saves you from having to directly print the newly created ofp to the console in order to get a preview for what it contains. If this code were bundled in a code chunk (such as in a Jupyter notebook), then only the final line of the code chunk would be printed.
The exact number of rows and columns that print will depend on the physical size of the REPL window. If you resize the console (e.g., in VS Code), Julia will adjust the number of rows/columns accordingly.
If you want to suppress the output, you can add a ; at the end of this statement, like this:
julia> ofp = dataset("Ecdat", "OFP"); # Nothing printsWe can use @glimpse() to find out the columns, data types, and peek at the first few values contained within the dataset.
julia> @glimpse(ofp)
Rows: 4406
Columns: 19
.OFP Int32 5, 1, 13, 16, 3, 17, 9, 3, 1, 0, 0, 44, 2, 1, 19,
.OFNP Int32 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0,
.OPP Int32 0, 2, 0, 5, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 1, 0, 0,
.OPNP Int32 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0,
.EMR Int32 0, 2, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
.Hosp Int32 1, 0, 3, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0,
.NumChron Int32 2, 2, 4, 2, 2, 5, 0, 0, 0, 0, 1, 5, 1, 1, 1, 0, 1,
.AdlDiff Int32 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1,
.Age Float64 6.9, 7.4, 6.6, 7.6, 7.9, 6.6, 7.5, 8.7, 7.3, 7.8,
.Black CategoricalValue{String, UInt8}yes, no, yes, no, no, no, no, no,
.Sex CategoricalValue{String, UInt8}male, female, female, male, female
.Married CategoricalValue{String, UInt8}yes, yes, no, yes, yes, no, no, no
.School Int32 6, 10, 10, 3, 6, 7, 8, 8, 8, 8, 8, 15, 8, 8, 12, 8
.FamInc Float64 2.881, 2.7478, 0.6532, 0.6588, 0.6588, 0.3301, 0.8
.Employed CategoricalValue{String, UInt8}yes, no, no, no, no, no, no, no, n
.Privins CategoricalValue{String, UInt8}yes, yes, no, yes, yes, no, yes, y
.Medicaid CategoricalValue{String, UInt8}no, no, yes, no, no, yes, no, no,
.Region CategoricalValue{String, UInt8}other, other, other, other, other,
.Hlth CategoricalValue{String, UInt8}other, other, poor, poor, other, pIf you're wondering why we need to place a @ at the beginning of the word so that it reads @glimpse() rather than glimpse(), that's because including a @ at the beginning denotes that this is a special type of function known as a macro. Macros have special capabilities in Julia, and many Tidier.jl functions that operate on data frames are implemented as macros. In this specific instance, we could have implemented @glimpse() without making use of any of the macro capabilities. However, for the sake of consistency, we have kept @glimpse() as a macro so that you can remember a basic rule of thumb: if Tidier.jl operates on a dataframe, then we will use macros rather than functions. The TidierPlots.jl package is a slight exception to this rule in that it is nearly entirely implemented as functions (rather than macros), and this will be described more in the TidierPlots documentation.
To avoid having to keep track of capitalization, data analysts often prefer column names to be in snake_case rather than TitleCase. Let's quickly apply this transformation to the ofp dataset.
julia> ofp = @clean_names(ofp)
julia> @glimpse(ofp)
Rows: 4406
Columns: 19
.ofp Int32 5, 1, 13, 16, 3, 17, 9, 3, 1, 0, 0, 44, 2, 1, 19,
.ofnp Int32 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0,
.opp Int32 0, 2, 0, 5, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 1, 0, 0,
.opnp Int32 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0,
.emr Int32 0, 2, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
.hosp Int32 1, 0, 3, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0,
.num_chron Int32 2, 2, 4, 2, 2, 5, 0, 0, 0, 0, 1, 5, 1, 1, 1, 0, 1,
.adl_diff Int32 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1,
.age Float64 6.9, 7.4, 6.6, 7.6, 7.9, 6.6, 7.5, 8.7, 7.3, 7.8,
.black CategoricalValue{String, UInt8}yes, no, yes, no, no, no, no, no,
.sex CategoricalValue{String, UInt8}male, female, female, male, female
.married CategoricalValue{String, UInt8}yes, yes, no, yes, yes, no, no, no
.school Int32 6, 10, 10, 3, 6, 7, 8, 8, 8, 8, 8, 15, 8, 8, 12, 8
.fam_inc Float64 2.881, 2.7478, 0.6532, 0.6588, 0.6588, 0.3301, 0.8
.employed CategoricalValue{String, UInt8}yes, no, no, no, no, no, no, no, n
.privins CategoricalValue{String, UInt8}yes, yes, no, yes, yes, no, yes, y
.medicaid CategoricalValue{String, UInt8}no, no, yes, no, no, yes, no, no,
.region CategoricalValue{String, UInt8}other, other, other, other, other,
.hlth CategoricalValue{String, UInt8}other, other, poor, poor, other, pNow that our column names are cleaned up, we can ask some basic analysis questions.
Because age is measured in decades according to the dataset documentation), we will multiply everyone's age by 10 before we calculate the median.
julia> @chain ofp @group_by(region) @summarize(mean_age = mean(age * 10))
4×2 DataFrame
Row │ region mean_age
│ Cat… Float64
─────┼───────────────────
1 │ other 73.987
2 │ midwest 74.0769
3 │ noreast 73.9343
4 │ west 74.1165Overall, the mean age looks pretty similar across regions. The fact that we were able to calculate each region's age also reveals that there are no missing values. Any region containing a missing value would have returned missing instead of a number.
The @chain macro, which is defined in the Chain package and re-exported by Tidier, allows us to pipe together multiple operations sequentially from left to right. In the example above, the ofp dataset is being piped into the first argument of the @group_by() macro, the result of which is then being piped into the @summarize() macro, which is then automatically removing the grouping and returning the result.
For grouped data frames, @summarize() behaves differently than other Tidier.jl macros: @summarize() removes one layer of grouping. Because the data frame was only grouped by one column, the result is no longer grouped. Had we grouped by multiple columns, the result would still be grouped by all but the last column. The other Tidier.jl macros keep the data grouped. Grouped data frames can be ungrouped using @ungroup(). If you apply a new @group_by() macro to an already-grouped data frame, then the newly specified groups override the old ones.
When we use the @chain macro, we are taking advantage of the fact that Julia macros can either be called using parentheses syntax, where each argument is separated by a comma, or they can be called with a spaced syntax where no parentheses are used. In the case of Tidier.jl macros, we always use the parentheses syntax, which makes is very easy to use the spaced syntax when working with @chain.
An alternate way of calling @chain using the parentheses syntax is as follows. From a purely stylistic perspective, I don't recommend this because it adds a number of extra characters. However, if you're new to Julia, it's worth knowing about this form so that you realize that there is no magic involved when working with macros.
julia> @chain(ofp, @group_by(region), @summarize(mean_age = mean(age * 10)))
4×2 DataFrame
Row │ region mean_age
│ Cat… Float64
─────┼───────────────────
1 │ other 73.987
2 │ midwest 74.0769
3 │ noreast 73.9343
4 │ west 74.1165On the other hand, either of these single-line expressions can get quite hard-to-read as more and more expressions are chained together. To make this easier to handle, @chain supports multi-line expressions using begin-end blocks like this:
julia> @chain ofp begin
@group_by(region)
@summarize(mean_age = mean(age * 10))
end
4×2 DataFrame
Row │ region mean_age
│ Cat… Float64
─────┼───────────────────
1 │ other 73.987
2 │ midwest 74.0769
3 │ noreast 73.9343
4 │ west 74.1165This format is convenient for interactive data analysis because you can easily comment out individual operations and view the result. For example, if we wanted to know the mean age for the overall dataset, we could simply comment out the @group_by() operation.
julia> @chain ofp begin
# @group_by(region)
@summarize(mean_age = mean(age * 10))
end
1×1 DataFrame
Row │ mean_age
│ Float64
─────┼──────────
1 │ 74.0241While Julia has a number of great data analysis packages, the most mature and idiomatic Julia package for data analysis is DataFrames.jl. Most other data analysis packages in Julia build on top of DataFrames.jl, and Tidier.jl is no exception.
DataFrames.jl emphasizes idiomatic Julia code without macros. While it is elegant, it can be verbose because of the need to write out anonymous functions. DataFrames.jl also emphasizes correctness, which means that errors are favored over warnings. For example, grouping by one variable and then subsequently grouping the already-grouped data frame by another variable results in an error in DataFrames.jl. These restrictions, while justified in some instances, can make interactive data analysis feel clunky and slow.
A number of macro-based data analysis packages have emerged as extensions of DataFrames.jl to make data analysis syntax less verbose, including DataFramesMeta.jl, Query.jl, and DataFrameMacros.jl. All of these packages have their strengths, and each of these served as an inspiration towards the creation of Tidier.jl.
What sets Tidier.jl apart is that it borrows the design of the tried-and-widely-adopted tidyverse and brings it to Julia. Our goal is to make data analysis code as easy and readable as possible. In our view, the reason you should use Tidier.jl is because of the richness, consistency, and thoroughness of the design made possible by bringing together two powerful tools: DataFrames.jl and the tidyverse. In Tidier.jl, nearly every possible transformation on data frames (e.g., aggregating, pivoting, nesting, and joining) can be accomplished using a consistent syntax. While you always have the option to intermix Tidier.jl code with DataFrames.jl code, Tidier.jl strives for completeness -- there should never be a requirement to fall back to DataFrames.jl for any kind of data analysis task.
Tidier.jl also focuses on conciseness. This shows up most readily in two ways: the use of bare column names, and an approach to auto-vectorizing code.
-
Bare column names: If you are referring to a column named
a, you can simply refer to it asain Tidier.jl. You are essentially referring toaas if it was within an anonymous function, where the variableawas mapped to the columnain the data frame. If you want to refer to an objectathat is defined outside of the data frame, then you can write!!a, which we refer to as "bang-bang interpolation." This syntax is motivated by the tidyverse, where the!!operator was selected because it is the least-bad "polite fiction" way of representing lazy interpolation. -
Auto-vectorized code: Most data transformation functions and operators are intended to be used on scalars. However, transformations are usually performed on columns of data (represented as 1-dimensional arrays, or vectors), which means that most functions need to be vectorized, which can get unwieldy and verbose. However, there are functions which operate directly on vectors and thus should not be vectorized when applied to columns (e.g.,
mean()andmedian()). Tidier.jl uses a customizable look-up table to know which functions to vectorize and which ones not to vectorize. This means that you can largely leave code as un-vectorized (i.e.,mean(a + 1)rather thanmean(a .+ 1)), and Tidier.jl will correctly infer convert the first code into the second before running it. There are several ways to manually override the defaults.
Lastly, the reason you should consider using Tidier.jl is that it brings a consistent syntax not only to data manipulation but also to plotting (by wrapping Makie.jl and AlgebraOfGraphics.jl) and to the handling of categorical variables, strings, and dates. Wherever possible, Tidier.jl uses existing classes rather than defining new ones. As a result, using Tidier.jl should never preclude you from using other Base Julia functions with which you may already be familiar.
If you're happy with the R tidyverse, then there's no imminent reason to switch to using Tidier.jl. While DataFrames.jl (the package on which TidierData.jl depends) is faster than R's dplyr and tidyr on benchmarks, there are other faster backends in R that allow for the use of tidyverse syntax with better speed (e.g., dtplyr, tidytable, tidypolars).
The primary reason to consider using Tidier.jl is the value proposition of using Julia itself. Julia has many similarities to R (e.g., interactive coding in a console, functional style, multiple dispatch, dynamic data types), but unlike R, Julia is automatically compiled (to LLVM) before it runs. This means that certain compiler optimations, which are normally only possible in more verbose languages like C/C++ become available to Julia. There are a number of situations in R where the end-user is able to write fast R code as a direct result of C++ being used on the backend (e.g., through the use of the Rcpp package). This is why R is sometimes referred to as a glue language -- because it provides a very nice way of glueing together faster C++ code.
The main value proposition of Julia is that you can use it as both a glue language and as a backend language. Tidier.jl embraces the glue language aspect of Julia while relying on packages like DataFrames.jl and Makie.jl on the backend.
While Julia has very mature backends, we hope that Tidier.jl demonstrates the value of, and need for, more glue-oriented data analysis packages in Julia.
Tidier comes with batteries included. If you are using Tidier, you generally won't have to load in other packages for basic data analysis. Tidier is meant for interactive use. You can start your code with using Tidier and expect to have what you need at your fingertips.
If you are a package developer, then you definitely should consider depending on one of the smaller packages that make up Tidier.jl rather than Tidier itself. For example, if you want to use the categorical variable functions from Tidier, then you should use rely on only TidierCats.jl as a dependency.
Either approach is okay. For most users, we recommend updating Tidier.jl directly, as this will update the underlying packages up to their latest minor versions (but not necessarily up to their latest patch release). However, if you need access to the latest functionality in the underlying packages, you should feel free to update them directly. We will keep Tidier.jl future-proof to underlying package updates, so this shouldn't cause any problems with Tidier.jl.
TidierData.jl is a package dedicated to data transformation and reshaping, powered by DataFrames.jl, ShiftedArrays.jl, and Cleaner.jl. It focuses on functionality within the dplyr, tidyr, and janitor R packages.
TidierPlots.jl is a package dedicated to plotting, powered by Makie.jl. It focuses on functionality within the ggplot2 R package.
TidierDB.jl is a package dedicated to data transformation on databases using syntax similar to TidierData. It focuses on functionality within the dbplyr R package.
TidierFiles.jl is a package dedicated to reading and writing tabular data. It focuses on functionality within the readr, haven, readxl, and writexl R packages.
TidierCats.jl is a package dedicated to handling categorical variables, powered by CategoricalArrays.jl. It focuses on functionality within the forcats R package.
TidierDates.jl is a package dedicated to handling dates and times. It focuses on functionality within the lubridate R package.
TidierStrings.jl is a package dedicated to handling strings. It focuses on functionality within the stringr R package.
TidierText.jl is a package dedicated to handling and tidying text data. It focuses on functionality within the tidytext R package.
TidierVest.jl is a package dedicated to scraping and tidying website data. It focuses on functionality within the rvest R package.
See News for the latest updates.
Is there a tidyverse feature missing that you would like to see in Tidier.jl? Please file a GitHub issue to start a discussion.