close

How to create dataframe from list in Spark SQL?

Hello Guys, How are you all? Hope You all Are Fine. Today We Are Going To learn about How to create dataframe from list in Spark SQL in Python. So Here I am Explain to you all the possible Methods here.

Without wasting your time, Let’s start This Article.

Table of Contents

How to create dataframe from list in Spark SQL?

  1. How to create dataframe from list in Spark SQL?

    i had to work with multiple columns and types – the example below has one string column and one integer column. A slight adjustment to Pushkr's code (above) gives:

  2. create dataframe from list in Spark SQL

    i had to work with multiple columns and types – the example below has one string column and one integer column. A slight adjustment to Pushkr's code (above) gives:

Method 1

here is how –

from pyspark.sql.types import *

cSchema = StructType([StructField("WordList", ArrayType(StringType()))])

# notice extra square brackets around each element of list 
test_list = [['Hello', 'world']], [['I', 'am', 'fine']]

df = spark.createDataFrame(test_list,schema=cSchema) 

Method 2

i had to work with multiple columns and types – the example below has one string column and one integer column. A slight adjustment to Pushkr’s code (above) gives:

from pyspark.sql.types import *

cSchema = StructType([StructField("Words", StringType())\
                      ,StructField("total", IntegerType())])

test_list = [['Hello', 1], ['I am fine', 3]]

df = spark.createDataFrame(test_list,schema=cSchema) 

output:

 df.show()
 +---------+-----+
|    Words|total|
+---------+-----+
|    Hello|    1|
|I am fine|    3|
+---------+-----+

Summery

It’s all About this issue. Hope all Methods helped you a lot. Comment below Your thoughts and your queries. Also, Comment below which Method worked for you? Thank You.

Also, Read