Skip to content

UnderstandingBigData

  • Blog
  • Spark Tutorial
    • Spark Dataframe
      • Topics1
        • SPARK DATAFRAME SELECT
        • SPARK FILTER FUNCTION
        • SPARK distinct and dropDuplicates
        • SPARK DATAFRAME Union AND UnionAll
        • Spark Dataframe withColumn
        • Spark Dataframe drop rows with NULL values
        • Spark Dataframe Actions
      • Topics2
        • How To Replace Null Values in Spark Dataframe
        • How to Create Empty Dataframe in Spark Scala
    • Spark Performance
      • Spark Lazy Evaluation
      • Spark Broadcast Variable explained
      • Repartition in SPARK
    • SparkSQL
      • Hive/Spark – Find External Tables in hive from a List of tables
      • Spark Read multiline (multiple line) CSV file with Scala
      • Spark Read JSON file
      • How to drop columns in dataframe using Spark scala
      • Spark Sql Inner Join
      • Spark SQL Count Function
    • Spark Externals
      • correct column order during insert into Spark Dataframe
      • Spark Function to check Duplicates in Dataframe
      • Spark UDF to Check Count of Nulls in each column
      • Spark Escape Double Quotes in Input File
    • Spark Practise
      • SPrac1
  • HDFS Tutorial
    • HDFS Replication Factor
    • HDFS Data Blocks and Block Size
  • Hive Tutorial
    • HiveLearning1
      • HIVE DATA TYPES
      • Hive Table Creation
      • HIVE ALTER TABLE
      • Hive Table Partition
      • Hive Split a row into multiple rows
      • HIVE SHOW PARTITIONS
    • HiveLearning2
      • Hive Insert Into vs Insert Overwrite
      • HIVE DROP TABLE
  • Scala Tutorial For Spark
    • ScalaLearning1
      • What is Functional Programming
      • SCALA TYPE INFERENCE
      • Scala Mutability vs Immutability
      • Scala Lazy Evaluation
      • Scala String Interpolation
      • Scala Pattern Matching
      • SCALA CLASS
      • SCALA SINGLETON AND COMPANION OBJECT
      • SCALA CASE CLASS
    • ScalaLearning2
      • SCALA FUNCTIONS
      • Scala Try Catch Finally
  • Azure Databricks
    • Databricks
      • Different ways of creating delta table in Databricks

Category: Uncategorized

Spark UDF to Check Count of Nulls in each column

In this blog we will create a Spark UDF to Check Count of Nulls in each column. There could be a scenario where we would […]

Uncategorized

Spark Function to check Duplicates in Dataframe

Here we will create a function to check if dataframe has duplicates Here we will not only create one method but will try and create […]

Uncategorized

Copyright © 2019 | All Rights Reserved

Shark Magazine by Shark Themes

 

Loading Comments...