This post attempts to continue the previous introductory series “Getting started with Spark in Python” with the topics UDFs and Window Functions.

• Part 1 Getting Started – covers basics on distributed Spark architecture, along with Data structures (including the old good RDD collections (!), whose use has been kind of deprecated by Dataframes)
• Part 2 intro to Dataframes
• Part 3 intro to UDFs and Window Functions 
• Part 4 unit testing in PySpark environments

Note: For this post I’m using Spark 1.6.1. There are some minor differences in comparison to the new coming Spark 2.0, such as using a SparkSession object to initialize the Spark Context, instead of HiveContext as I do here. Nonetheless, the important parts are common in both.

Continue reading “Getting Started with Spark (part 3) – UDFs & Window functions” →