Can I read CSV with columns that has specific value(s) using Pandas?

Question

I want to read a CSV using Pandas but only certain columns and only rows with spicific values. for example I have a csv of "people and their heights", I want to read the "name" column and "height" column of people that are > "160cm" height only. I want to do this in the first step of read_csv() not after loading it.

import pandas as pd
cols = ['name','height']
df = pd.read_csv("people_and_heights.csv", usecols=cols)

so I want to add a condition to read rows with certain values only or rows that doesn't have nulls for example.

Are you doing this for the purpose of reducing memory usage, or some other reason? — Nick ODell
– Nick ODell, Commented Jan 16, 2022 at 21:39
If this is a memory issue as @NickODell is suggesting, your best bet is probably to load the .csv in chunks. I don't think you can avoid loading the rows you don't want as they will need to be loaded to check if they meet your criteria. — Pepsi-Joe
– Pepsi-Joe, Commented Jan 16, 2022 at 21:49
exactly @NickODell . I'm thinking to overcome memory issues as I'm dealing with more than 50GB zipped CSV's and many of the rows are useless since it's null or with values that are byonde the scope of my analysis. — rfengineer
– rfengineer, Commented Jan 16, 2022 at 22:01
@Pepsi-Joe that would be my last resort really, especially with many CSV's to deal with. — rfengineer
– rfengineer, Commented Jan 16, 2022 at 22:05

Can · Accepted Answer · 2022-01-16 22:45:11Z

0

How about this?:

import pandas as pd
from io import StringIO

with open("people_and_heights.csv") as file:
    colNames = "\"col1\",\"name\",\"col3\",\"height\""
    filteredCsv = "\n".join([colNames,"".join([line for index,line in enumerate(file) if index != 0 and int(line.split(',')[3]) >= 165])])

df = pd.read_csv(StringIO(filteredCsv),usecols=["name","height"])

edited Jan 16, 2022 at 22:45

answered Jan 16, 2022 at 22:28

Can

11 bronze badge

Sign up to request clarification or add additional context in comments.

1 Comment

Can Over a year ago

ah right :) I only thought about "I want to do this in the first step of read_csv() not after loading it." part. I didn't notice the comments sorry :)

Collectives™ on Stack Overflow

Can I read CSV with columns that has specific value(s) using Pandas?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related