Today we will learn how to create empty dataframe in Spark Scala. We will cover various methods on how to create empty dataframe with no […]
Category: Spark Dataframe
How To Replace Null Values in Spark Dataframe
In Previous chapter we learned about Spark Dataframe Actions and today lets check out How to replace null values in Spark Dataframe. It is really important to handle […]
Spark Dataframe Actions
When we call an Action on a Spark dataframe all the Transformations gets executed one by one. This happens because of Spark Lazy Evaluation which […]
Spark Dataframe drop rows with NULL values
The data we normally deal with may not be clean. In such cases we may need to clean the data by applying some logic . […]
Spark Dataframe withColumn
Using Spark withColumn() function we can add , rename , derive, split etc a Dataframe Column. There are many other things which can be achieved […]
SPARK DATAFRAME Union AND UnionAll
Using Spark Union and UnionAll you can merge data of 2 Dataframes and create a new Dataframe. Remember you can merge 2 Spark Dataframes only […]
SPARK distinct and dropDuplicates
Both Spark distinct and dropDuplicates function helps in removing duplicate records. One additional advantage with dropDuplicates() is that you can specify the columns to be […]
SPARK FILTER FUNCTION
Using Spark filter function you can retrieve records from the Dataframe or Datasets which satisfy a given condition. People from SQL background can also use […]
SPARK DATAFRAME SELECT
Today we will learn how to Select columns from a Spark Dataframe. While selecting we can show complete list of columns or select only few […]
Show full column content of Spark Dataframe
When we do a dataframe.show() , it does now show full column content. It shows only 20 records which is the default number of rows […]