close

How do I release memory used by a pandas dataframe?

Hello Guys, How are you all? Hope You all Are Fine. Today We Are Going To learn about How do I release memory used by a pandas dataframe in Python. So Here I am Explain to you all the possible Methods here.

Without wasting your time, Let’s start This Article.

Table of Contents

How do I release memory used by a pandas dataframe?

  1. How do I release memory used by a pandas dataframe?

    As noted in the comments, there are some things to try: gc.collect may clear stuff, for example. At least from my experience, these things sometimes work and often don't.

  2. release memory used by a pandas dataframe

    As noted in the comments, there are some things to try: gc.collect may clear stuff, for example. At least from my experience, these things sometimes work and often don't.

Method 1

As noted in the comments, there are some things to try: gc.collect may clear stuff, for example. At least from my experience, these things sometimes work and often don’t.

There is one thing that always works, however, because it is done at the OS, not language, level.

Suppose you have a function that creates an intermediate huge DataFrame, and returns a smaller result (which might also be a DataFrame):

def huge_intermediate_calc(something):
    ...
    huge_df = pd.DataFrame(...)
    ...
    return some_aggregate

Then if you do something like

import multiprocessing

result = multiprocessing.Pool(1).map(huge_intermediate_calc, [something_])[0]

Then the function is executed at a different process. When that process completes, the OS retakes all the resources it used. There’s really nothing Python, pandas, the garbage collector, could do to stop that.

Method 2

This solves the problem of releasing the memory for me!!!

import gc
import pandas as pd

del [[df_1,df_2]]
gc.collect()
df_1=pd.DataFrame()
df_2=pd.DataFrame()

the data-frame will be explicitly set to null

in the above statements

Firstly, the self reference of the dataframe is deleted meaning the dataframe is no longer available to python there after all the references of the dataframe is collected by garbage collector (gc.collect()) and then explicitly set all the references to empty dataframe.

Summery

It’s all About this issue. Hope all Methods helped you a lot. Comment below Your thoughts and your queries. Also, Comment below which Method worked for you? Thank You.

Also, Read