Java

  • Setting up sparklyr

    The sparklyr package allows you to work with data sets, which are too large for the memory of your computer, by using the well-known verbs of dplyr on Apache Spark. However, before you can write a single line of such code, your machine has to be prepared for it. This is a bit more involved than the installation of an ordinary R package, because Spark is not written in R. It runs on the Java virtual machine and, therefore, needs a working Java installation as well as a copy of Spark itself. This post goes through those steps one by one.