Checking file type with django form: 'application/octet-stream' issue

As a follow up to Liyosi answer, I also used python-magic. There seems to be a bug with libmagic where it still incorrectly identifies some files as application/octet-stream Described better on the code

    def _handle509Bug(self, e):
        # libmagic 5.09 has a bug where it might fail to identify the
        # mimetype of a file and returns null from magic_file (and
        # likely _buffer), but also does not return an error message.
        if e.message is None and (self.flags & MAGIC_MIME):
            return "application/octet-stream"
        else:
            raise e

To get around this issue, I had to instantiate a magic object and make use of uncompressed and mime attributes. To complete Liyosi example:

import magic
from django.core.exceptions import ValidationError

def validate_file_type(upload):
    allowed_filetypes = [
        'application/pdf', 'image/jpeg', 'image/jpg', 'image/png',
        'application/msword']
    validator = magic.Magic(uncompress=True, mime=True)
    file_type = validator.from_buffer(upload.read(), mime=True)
    if file_type not in allowed_filetypes:
        raise ValidationError('Unsupported file')

The most fool proof way of telling is by snooping into the file contents by reading its metadata in the file header.

In most files, this file header is usually stored at the beginning of the file, though in some, it may be located in other locations.

python-magic helps you to do this, but the trick is to always reset the pointer at the beginning of the file, before trying to guess its mime type, else you will sometimes be getting appliation/octet-stream mime type if the reader's pointer has advanced past the file header location to other locations that just contains arbitrary stream of bytes.

For example, if you have a django validator function that tries to validate uploaded files for mime types:

import magic
from django.core.exceptions import ValidationError

def validate_file_type(upload):
    allowed_filetypes = [
        'application/pdf', 'image/jpeg', 'image/jpg', 'image/png',
        'application/msword']
    upload.seek(0)
    file_type = magic.from_buffer(upload.read(1024), mime=True)
    if file_type not in allowed_filetypes:
        raise ValidationError(
            'Unsupported file')

You should not rely on the MIME type provided, but rather the MIME type discovered from the first few bytes of the file itself.

This will help eliminate the generic MIME type issue.

The problem with this approach is that it will usually rely on some third party tool (for example the file command commonly found on Linux systems is great; use it with -b --mime - and pass in the first few bytes of your file to have it give you the mime type).

The other option you have is to accept the file, and try to validate it by opening it with a library.

So if pypdf cannot open the file, and the built-in zip module cannot open the file, and rarfile cannot open the file - its most likely something that you don't want to accept.