How to remove specific lines from a text file, using Python

Question

I have a text file with data in it. I am trying to create a python code that will format this data in a particular way so another code I have can read it as an input. So, I am trying to remove specific lines and columns, etc. The text file contains some information at the top of the file, and then columns and columns of numerical data below it. However, I only want the numerical data. I do not want the first 34 lines of other information written in the text file.

So, my current question is: How can I remove specific lines of a text file? And how do I print this result so I can see if it worked or not?

Before you come at me: Yes, I have looked up my question on StackOverflow. I still don't get it, and I don't know what I'm doing wrong. I need your help! :)

(Image of my text file for reference)

Since each of the lines I want to remove in the text file begin with a '#', maybe it would be possible to tell the code to remove all lines that begin with "#'. Or maybe I should just specify I want the first 34 lines deleted. Either way, I'm confused on how to do this.

Separated below are all my different attempts: (Don't worry, I have imported pandas for my attempts, it's just not written here.)

with open('EBSD_data.txt','r') as f:
    lines = f.readlines() 
with open('EBSD_data.txt','w') as f:
     for lines in lines:
    if line.strip("\n").startswith('#'):
      f.write(lines)

with open('EBSD_data.txt', 'r') as E:
    data = E.read().splitlines(True)
with open('EBSD_data.txt', 'w') as EB:
    EB.writelines(data[34:])
    print(EB.read)

with open('EBSD_data.txt', 'w') as data: 
  for lines in data:
    if not lines.startswith('#'):
      data.write(lines)
with open('EBSD_data.txt', 'r') as data2:
     print(data2)

x = open('EBSD_data.txt')
for line in x:
  if line.startswith('#'):
    del(line)
x.head()

lines = []
with open(r'EBSD_data.txt','r') as fp:
  lines = fp.readlines()
with open(r'EBSD_data.txt','w') as data:
 for number, line in enumerate(lines):
  if number not in [34]:
    fp.write(line)

With some of these attempts, it'll look like it runs fine, but then I have issues with printing my results. The typical error I am encountering after trying to print my result is:

<_io.TextIOWrapper name='EBSD_data.txt' mode='w' encoding='UTF-8'>

Any guidance you could give would be great! Thank you! :)

Welcome to Stack Overflow! As you have tagged this question with pandas, I can't help but suggest using the read_csv pandas function: df = pd.read_csv(skiprows=34, header=None) pandas.pydata.org/docs/reference/api/pandas.read_csv.html — Peter Leimbigler
– Peter Leimbigler, Commented Apr 11, 2022 at 14:21
@CaptainCaveman, my first attempt for example gives me that error. — Brienne of Tarth
– Brienne of Tarth, Commented Apr 11, 2022 at 14:24
@PeterLeimbigler, I just edited my post and removed the 'pandas' tag so maybe you are able to help now? — Brienne of Tarth
– Brienne of Tarth, Commented Apr 11, 2022 at 14:25

Niels Henkens · Accepted Answer · 2022-04-11 14:25:30Z

1

I personally would always write to a new file and keep the original one...\

This should work:

with open('EBSD_data.txt', 'r') as data: 
  with open('EBSD_data_filtered.txt', 'w') as outfile:
      for line in data:
          if not line.startswith('#'):
              outfile.write(line)

with open('EBSD_data_filtered.txt', 'r') as data2:
    for line in data2:
        print(line)

answered Apr 11, 2022 at 14:25

Niels Henkens

2,7161 gold badge14 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Brienne of Tarth Over a year ago

Thanks for your help! This way worked well! :)

Peter Leimbigler · Accepted Answer · 2022-04-11 14:26:01Z

0

Instead of manually reading each line with a Python context manager, you can use read_csv, skipping the first 34 rows of the file (skiprows=34), and without reading any column names from the file (header=None):

import pandas as pd

df = pd.read_csv('EBSD_data.txt', skiprows=34, header=None)

# print first five rows to confirm expected result
print(df.head())

# Optional: write result back to a CSV file, without the default 
# pandas integer index
df.to_csv('EBSD_data_filtered.txt', index=None)

answered Apr 11, 2022 at 14:26

Peter Leimbigler

11.1k1 gold badge27 silver badges39 bronze badges

Collectives™ on Stack Overflow

How to remove specific lines from a text file, using Python

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related