How to pass string via STDIN into terminal command being executed within python script?

Question

I need to generate postgres schema from a dataframe. I found csvkit library to come closet to matching datatypes. I can run csvkit and generate postgres schema over a csv on my desktop via terminal through this command found in docs:

csvsql -i postgresql myFile.csv

csvkit docs - https://csvkit.readthedocs.io/en/stable/scripts/csvsql.html

And I can run the terminal command in my script via this code:

import os
a=os.popen("csvsql -i postgresql Desktop/myFile.csv").read()

However I have a dataframe, that I have converted to a csv string and need to generate schema from the string like so:

csvstr = df.to_csv()

In the docs it says that under positional arguments:

The CSV file(s) to operate on. If omitted, will accept
                        input on STDIN

How do I pass my variable csvstr into the line of code a=os.popen("csvsql -i postgresql csvstr").read() as a variable?

I tried to do the below line of code but got an error OSError: [Errno 7] Argument list too long: '/bin/sh':

a=os.popen("csvsql -i postgresql {}".format(csvstr)).read()

Thank you in advance

Nadav Ruskin · Accepted Answer · 2018-07-15 14:06:05Z

2

You can't pass such a big string via commandline! You have to save the data to a file and pass its path to csvsql.

import csv

csvstr = df.to_csv()
with open('my_cool_df.csv', 'w', newline='') as csvfile:
    csvwriter= csv.writer(csvfile)
    csvwriter.writerows(csvstr)

And later:

a=os.popen("csvsql -i postgresql my_cool_df.csv")

answered Jul 15, 2018 at 14:06

Nadav Ruskin

2923 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

RustyShackleford Over a year ago

I can already do this, and write dataframe directly to csv using `to_csv('file/path'). I would really like to pass a string so I can take in other data sources and make process more efficient.

RustyShackleford Over a year ago

do you think I could pass a string in this way? stackoverflow.com/questions/163542/…

Nadav Ruskin Over a year ago

To be honest- I'm not sure. Refer to csvkit.readthedocs.io/en/1.0.2/… on syntax to passing STDIN to csvkit. An stdin channel is different from an argument list, I don't think you should be hitting a limit. I'd test for performance though, I'm not sure Python can handle this faster than simply saving and loading a file will be.

Collectives™ on Stack Overflow

How to pass string via STDIN into terminal command being executed within python script?

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related