Python thinks a file is empty when opened in binary read

Question

I'm running Python 3.5.1 on Windows. I am attempting to find duplicate source code files in a directory by computing their hash. The problem is that Python seems to think some files are empty. Here is the relevant code snippet:

with open(path, 'rb') as afile:
    hasher = hashlib.md5()
    data = afile.read()
    hasher.update(data)
    print("len(data): {}, Path: {}, Hash:{}".format(len(data), path, hasher.hexdigest()))

Here is some example output:

len(data): 0, Path: h:\t\TCPServerSocket.h, Hash:d41d8cd98f00b204e9800998ecf8427e
len(data): 0, Path: h:\t\TCPSocket.cpp, Hash:d41d8cd98f00b204e9800998ecf8427e
len(data): 0, Path: h:\t\TCPSocket.h, Hash:d41d8cd98f00b204e9800998ecf8427e
len(data): 5073, Path: h:\t\ConfigFile.cpp, Hash:6188d6a0e0bc02edf27ce232689beff6

I assure you that these files are not empty, and Python is not throwing any errors during execution. Any ideas?

the path has the wrong slashes (Windows!) so stuff might get escaped. Do you use the os.path functions? — Henno Brandsma
– Henno Brandsma, Commented May 26, 2016 at 21:30
Hi, yes, I am using the os.path functions. Python is accessing the files fine, it just thinks that they are empty. I can open the files in an editor without issue as well. — user2145843
– user2145843, Commented May 26, 2016 at 21:33
Are you sure that is the code you're actually running? Your print statement has data: , but the output is len(data): . — John Gordon
– John Gordon, Commented May 26, 2016 at 21:46
Hi John. Yes, it runs. Copying the code and output to SO got out of sync. I have updated the post to be accurate. — user2145843
– user2145843, Commented May 27, 2016 at 13:19
I notice that is the md5 for an empty string, so go back before that. Check existence and size of the "path" variable before you open, maybe? — Kenny Ostrom
– Kenny Ostrom, Commented May 27, 2016 at 13:34

Tadhg McDonald-Jensen · Accepted Answer · 2016-05-27 13:54:27Z

2

I'll just delete this answer if it is not the case, but it's something you need to check. Put this directly before the open block

print("the path is {!r}".format(path))
print("path exists: ", os.path.exists(path))
print("it is a file: ", os.path.isfile(path))
print("file size is: ", os.path.getsize(path))

Because everything in your output is consistent with that file actually being empty. So maybe it is? My first thought was you might be zeroing out the file elsewhere, although you would figure that out pretty quickly.

edited May 27, 2016 at 13:54

Tadhg McDonald-Jensen

21.6k5 gold badges40 silver badges69 bronze badges

answered May 27, 2016 at 13:52

Kenny Ostrom

5,9913 gold badges26 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user2145843 Over a year ago

file size is: 0 for these files. I went back, erased the source, re-checked out the files, and re-loaded the files in my editor. Files appeared normal. Ran some other scripts in the tool chain and, wa la, the files are zero'd out. Yet, the editor retains the original file even upon reopening. I truly do not understand this behavior, but thank you for helping me to diagnose it.

Kenny Ostrom Over a year ago

The editor probably has auto-restore from a temp file. Look for opening those files elsewhere with the wrong flags (write instead of read). Easy mistake to make.

Greg Hilston · Accepted Answer · 2016-05-26 21:02:40Z

-1

I think you should computer the hash by calling hashlib.md5 on the files them self

import hashlib
hashlib.md5("filename").hexdigest()

Let me know if that continues to suggest files are empty

answered May 26, 2016 at 21:02

Greg Hilston

2,4242 gold badges26 silver badges36 bronze badges

1 Comment

user2145843 Over a year ago

I think your code only hashes the file name, but I want to hash the file contents (sorry if that was unclear).

Collectives™ on Stack Overflow

Python thinks a file is empty when opened in binary read

2 Answers 2

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related