According to the Python 3 documentation for built-in functions, ord() returns an integer:
Given a string representing one Unicode character, return an integer representing the Unicode code point of that character. For example, ord('a') returns the integer 97 [...]
This integer is represented by a sequence of bits, and so it is a valid operand to the bitwise XOR operator.
Edit
Python interprets the value 97 as an integer. And, internally, the integer numeric type is represented as a sequence of bits. The integer type is described in Numeric Types. Another section of that same documentation explains Bitwise Operations on Integer Types, and it states that
Bitwise operations only make sense for integers.
That is, bitwise operations on any other type, including a string, are invalid in Python.
So ord() returns an integer. Which is represented internally as a sequence of bits. Although its type is not explicitly "binary," Python still defines the bitwise XOR operator for integers.
To be clear, all values are represented internally as a sequence of bits. It's just that Python only defines the bitwise XOR operator for integers.