close

How to export a table dataframe in PySpark to csv?

Hello Guys, How are you all? Hope You all Are Fine. Today We Are Going To learn about How to export a table dataframe in PySpark to csv in Python. So Here I am Explain to you all the possible Methods here.

Without wasting your time, Let’s start This Article.

Table of Contents

How to export a table dataframe in PySpark to csv?

  1. How to export a table dataframe in PySpark to csv?

    For Apache Spark 2+, in order to save dataframe into single csv file. Use following command

  2. .export a table dataframe in PySpark to csv

    For Apache Spark 2+, in order to save dataframe into single csv file. Use following command

Method 1

If data frame fits in a driver memory and you want to save to local files system you can convert Spark DataFrame to local Pandas DataFrame using toPandas method and then simply use to_csv:

df.toPandas().to_csv('mycsv.csv')

Otherwise you can use spark-csv:

  • Spark 1.3df.save('mycsv.csv', 'com.databricks.spark.csv')
  • Spark 1.4+df.write.format('com.databricks.spark.csv').save('mycsv.csv')

In Spark 2.0+ you can use csv data source directly:

df.write.csv('mycsv.csv')

Method 2

For Apache Spark 2+, in order to save dataframe into single csv file. Use following command

query.repartition(1).write.csv("cc_out.csv", sep='|')

Here 1 indicate that I need one partition of csv only. you can change it according to your requirements.

Conclusion

It’s all About this issue. Hope all Methods helped you a lot. Comment below Your thoughts and your queries. Also, Comment below which Method worked for you? Thank You.

Also, Read