close

How to import a text file on AWS S3 into pandas without writing to disk

Hello Guys, How are you all? Hope You all Are Fine. Today We Are Going To learn about How to import a text file on AWS S3 into pandas without writing to disk in Python. So Here I am Explain to you all the possible Methods here.

Without wasting your time, Let’s start This Article.

Table of Contents

How to import a text file on AWS S3 into pandas without writing to disk?

  1. How to import a text file on AWS S3 into pandas without writing to disk?

    pandas uses boto for read_csv, so you should be able to:
    import boto data = pd.read_csv('s3://bucket....csv')

  2. import a text file on AWS S3 into pandas without writing to disk

    pandas uses boto for read_csv, so you should be able to:
    import boto data = pd.read_csv('s3://bucket....csv')

Method 1

pandas uses boto for read_csv, so you should be able to:

import boto
data = pd.read_csv('s3://bucket....csv')

If you need boto3 because you are on python3.4+, you can

import boto3
import io
s3 = boto3.client('s3')
obj = s3.get_object(Bucket='bucket', Key='key')
df = pd.read_csv(io.BytesIO(obj['Body'].read()))

Method 2

With s3fs it can be done as follow:

import s3fs
import pandas as pd
fs = s3fs.S3FileSystem(anon=False)

# CSV
with fs.open('mybucket/path/to/object/foo.pkl') as f:
    df = pd.read_csv(f)

# Pickle
with fs.open('mybucket/path/to/object/foo.pkl') as f:
    df = pd.read_pickle(f)

Summery

It’s all About this issue. Hope all Methods helped you a lot. Comment below Your thoughts and your queries. Also, Comment below which Method worked for you? Thank You.

Also, Read