0

i'm having a difficult time understanding what the second 'with open' function does here.

so, in the first 'with open' part, we've essentially said out = open(save_as_file, 'wb+') , right? (still new to using 'with open'). we later write to it and then 'with open' automatically closes the 'out' file.That part i get - we're writing this response object from Requests as a binary in a specified save_as_file location until we hit the 81920th character aka our buffer #.

what's going on in the second 'with open'? breaking it down the same way as above, it's pretty much fp = open(save_as_file, 'r') , right? What does that make fp, which was already assigned the request response object earlier? We're just opening the save_as_file to use it for reading but not reading or extracting anything from it, so I don't see the reason for it. If someone could explain in english just what's taking place and the purpose of the second 'with open' part, that would be much appreciated.

(don't worry about the load_from_file function at the end, that's just another function under the class)

def load_from_url(self, url, save_as_file=None):

    fp = requests.get(url, stream=True,
                      headers={'Accept-Encoding': None}).raw

    if save_as_file is None:
        return self.load_from_file(fp)

    else:
        with open(save_as_file, 'wb+') as out:
            while True:
                buffer = fp.read(81920)
                if not buffer:
                    break
                out.write(buffer)
        with open(save_as_file) as fp:
            return self.load_from_file(fp)
Improve this question
1

4 Answers 4

Reset to default
2

I'm the original author of the code that you're referring to; I agree it's a bit unclear.

If we hit the particular code at the else statement, this means that we want to save the data that we originally get from calling the URL to a file. Here, fp is actually the response text from the URL call.

We'll hit that else statement if, when ran from the command line, we pass in --cpi-file=foobar.txt and that file it doesn't actually exist yet; it acts as a target file as mentioned here. If you don't pass in --cpi-file=foobar.txt, then the program will not write to a file, it will just go straight to reading the response data (from fp) via load_from_file.

So then, if that file does not exist but we did pass it in the command line, we will grab data from the URL (fp), and write that data to the target file (save_as_file). It now exists for our reference (it will be on your file system), if we want to use it again in this script.

Then, we will open that exact file again and call load_from_file to actually read and parse the data that we originally got from the response (fp).

Now - if we run this script two times, both with --cpi-file=foobar.txt and foobar.txt doesn't exist yet, the first time the script runs, it will create the file and save the CPI data. The second time the script runs, it will actually avoid calling the CPI URL to re-downloaded the data again, and just go straight to parsing the CPI data from the file.

load_from_file is a bit of a misleading name, it should probably be load_from_stream as it could be reading the response data from our api call or from a file.

Hopefully that makes sense. In the next release of newcoder.io, I'll be sure to clear this language & code up a bit.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

You are correct that the second with statement opens the file for reading.

What happens is this:

  1. Load the response from the URL
  2. If save_as_file is None:
    1. Call load_from_file on the response and return the result
  3. Else:
    1. Store the contents of the response to save_as_file
    2. Call load_from_file on the contents of the file and return the result

So essentialy, if save_as_file is set it stores the response body in a file, processes it and then returns the processed result. Otherwise it just processes the response body and returns the result.

The way it is implemented here is likely because load_from_file expects a file-like object and the easiest way the programmer saw of obtaining that was to read the file back.

It could be done by keeping the response body in memory and using Python 3's io module or Python 2's StringIO to provide a file-like object that uses the response body from memory, thereby avoiding the need to read the file again.

fp is reassigned in the second with statement in the same way as any other variable would be if you assigned it another value.

Improve this answer

Comments

1

I tried with below code to simulate your case:

fp = open("/Users/example1.py",'wb+')

print "first fp",fp

with open("/Users/example2.py") as fp:
    print "second fp",fp

The output is:

first fp <open file '/Users/example1.py', mode 'wb+' at 0x10b200390>
second fp <open file '/Users/example2.py', mode 'r' at 0x10b200420>

So second fp is a local variable inside with block.

Your code seem want to first read data from the URL, and write it to save_as_file, and then read data from save_as_file again and do something with load_from_file, like validating the content.

Improve this answer

2 Comments

a good answer too - just liked the way @Raniz explained it to me like I was a 5 year old.
If you feel useful, consider a upvote:) My experience is adding break points will help you understand.
0

Here is a piece of code that describe it:

  1. __with__ provides a block that "cleans up" when existed
  2. Can handle exceptions that occur within the block
  3. Can also execute code when entered
class MyClass(object):

    def __enter__(self):
        print("entering the myclass %s")
        return self

    def __exit__(self, type, value, traceback):
        print("Exitinstance %s" %(id(self)))
        print("error type {0}".format(type))
        print("error value {0}".format(value))
        print("error traceback {0}".format(traceback))
        print("exiting the myclass")

    def sayhi(self):
        print("Sayhi instance %s" %(id(self)))

with MyClass() as cc:
    cc.sayhi()


print("after the block ends")
Improve this answer

1 Comment

you can paste the code in pythontutor.com and see how it works

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.