The sparklyr package allows you to work with data sets, which are too large for the memory of your computer, by using the well-known verbs of dplyr on Apache Spark. However, before you can write a single line of such code, your machine has to be prepared for it. This is a bit more involved than the installation of an ordinary R package, because Spark is not written in R. It runs on the Java virtual machine and, therefore, needs a working Java installation as well as a copy of Spark itself. This post goes through those steps one by one.
The cleaning and transformation of data belong to the most time consuming parts of any economic analysis. As long as a data set fits into the memory of your computer, packages like dplyr or data.table are all you need to get this job done. But once it does not, R alone reaches its limits, because it holds all its objects in memory. This is where Apache Spark comes into play. Spark is a computing engine, which distributes both the data and the calculations over the machines of a cluster – or over the cores of a single computer. The sparklyr package is the interface between R and Spark. Its most attractive feature for R users is that it translates the well-known verbs of dplyr into Spark SQL. This means that you can keep writing the code you already know and let it run on data which would never fit into your RAM. In the following I use the same example as in my introductions to dplyr and data.table, so that the three approaches can be compared directly.