how to update a variable inside loop in python?

Question

I'm trying to get data from an api where the offset is not known.so I've to iterate over until the data is not null and in each offset there is 10 record so incrementing by 10 will next 10 record .

2.since the data is huge i want to write data in different files. For that i want that after each 500 offset it write's the data in the the next file. If I've 1300 offset in total i want that it writes the 0-500 to file1 510-1000 to file2 1010-1350 to file3

i=0
data = ''
while data != ',': 
     url = test.api?offset{i}
     response=requests.get(url)
     data=response.text
     data+=data
     if i%500==0:
         fo=open("sample.txt")
         fo.write(data)
         i+=10

First observation: use with in place of fo=open(). the code will be cleaner and gurantee closing the files. — darren
– darren, Commented Oct 12, 2022 at 11:51
fo = open(f"sample_{file_number}.text","w") this will open a file named sample_1.text then sample_2.text etc... but file_number must be set to 0 before while and incremented inside if i%500 ==0: — A. Herlas
– A. Herlas, Commented Oct 12, 2022 at 11:53

jack · Accepted Answer · 2022-10-12 12:55:54Z

1

Try this:

i = 0
data = ''
j = 1
while data !=',':
    if i%500==0:
        with open(f"file{j}.txt", "w") as fo:
            fo.write(data)
        j += 1
        data = ''
    url = f"test.api?offset{i}"
    response = requests.get(url)
    data += response.text
    i+=10

if data != '':
    with open(f"file{j}.txt", "w") as fo:
        fo.write(data)

Basically I created another variable j to keep track of the file name, and the data variable is reset to an empty string every time after writing to the file. Also like D.L mentioned, using the with keyword would allow the file to close automatically after writing, resulting in cleaner syntax.Also, I am assuming you are developing with Python3 so I used f-string syntax.

From what I can gather, I think the data variable shouldn't be assigned and then added to itself every iteration, as that will cause a duplication of data obtained from the current iteration's get request, without preserving the data from the get requests of all previous iterations. I assume you would want to keeping adding more data until the offset reaches a multiple of 500, so that's what I tried to do.

edited Oct 12, 2022 at 12:55

answered Oct 12, 2022 at 11:53

jack

763 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

david01 Over a year ago

but this will not work for the case where i is not divisible by 500,like where offset in 1000-1300

jack Over a year ago

ah in that case we can just add one last write to file for the remaining data after the while loop has ended! I'll edit my answer to include that.

Omer Glazer · Accepted Answer · 2022-10-12 12:14:49Z

0

import requests

i: int = 0
data: str = ""

while payload != ",":
    url: str = f"test.api?offset{i}"
    response: requests.Response = requests.get(url)
    payload: str = response.text
    data += payload
    if i != 0 and i % 500 == 0:
        with open(f"offset_{i-500}-{i}.txt", "w+") as f:
            f.write(data)
        data = ""
    i += 10

a solution that complies with python conventions,

plus you need to verify i != 0, since 0 % 500 == 0 will result True.

edited Oct 12, 2022 at 12:14

answered Oct 12, 2022 at 12:07

Omer Glazer

365 bronze badges

Collectives™ on Stack Overflow

how to update a variable inside loop in python?

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related