Introduction to sparklyr

Sat, 05 Sep 2026 00:00:00 +0000

The cleaning and transformation of data belong to the most time consuming parts of any economic analysis. As long as a data set fits into the memory of your computer, packages like dplyr or data.table are all you need to get this job done. But once it does not, R alone reaches its limits, because it holds all its objects in memory. This is where Apache Spark comes into play. Spark is a computing engine, which distributes both the data and the calculations over the machines of a cluster – or over the cores of a single computer. The sparklyr package is the interface between R and Spark.¹ Its most attractive feature for R users is that it translates the well-known verbs of dplyr into Spark SQL. This means that you can keep writing the code you already know and let it run on data which would never fit into your RAM. In the following I use the same example as in my introductions to dplyr and data.table, so that the three approaches can be compared directly.

Sparklyr on r-econometrics

Introduction to sparklyr