When we do a dataframe.show() , it does now show full column content. It shows only 20 records which is the default number of rows which is displayed with show(). Also it does not show the complete column content.
To see the entire data we need to pass parameter
show(number of records , boolean value)
number of records : The number of records you need to display. Default is 20.
boolean value : false means show complete column value. Default is true.
scala> df.show(3,false) +-------------------------+ | col| +-------------------------+ |2099-12-32 05:16:23:00:01| |2099-12-32 06:18:23:00:01| |2099-12-32 09.54:23:00:01| +-------------------------+
You now see that the number of records got reduced to 3 and also complete column value is visible. So now you know how to properly display the content of a dataframe in Spark.
Your can also write df.show(false) in which case it will show full column of 20 records.
Table of Contents
Show Full Column Contents for All Records
If you need to show the full column content for all records then you first need to find the count of records using df.count and store the value in a variable and then pass it inside show. Instead of this we can do as below
val df1 = Seq(("Smithxxxxxxxxxxxxxxxxxxxxxxxxxxx",23),("Rashmi",27)).toDF("Name","Age")
df1.show(df1.count().toInt,false)
+--------------------------------+---+
|Name |Age|
+--------------------------------+---+
|Smithxxxxxxxxxxxxxxxxxxxxxxxxxxx|23 |
|Rashmi |27 |
+--------------------------------+---+
Conclusion
Today we learn how to show full column content of spark dataframe.
🙂 kudos for learning something new 🙂
