Why would it return "\x21"? The \x notation is reserved for non-printable characters, but since \x21 is equivalent to ! then that's what's displayed.
The inspect version of a string is generally the most readable version of it, not the most literal.
Likewise:
"!".ord
# => 33
"\x21".ord
# => 33
"\x21".ord.to_s(16)
# => "21"
33.chr
# => "!"
There's a number of special characters documented in the Strings section:
\a bell, ASCII 07h (BEL)
\b backspace, ASCII 08h (BS)
\t horizontal tab, ASCII 09h (TAB)
\n newline (line feed), ASCII 0Ah (LF)
\v vertical tab, ASCII 0Bh (VT)
\f form feed, ASCII 0Ch (FF)
\r carriage return, ASCII 0Dh (CR)
\e escape, ASCII 1Bh (ESC)
\s space, ASCII 20h (SPC)
\\ backslash, \
\nnn octal bit pattern, where nnn is 1-3 octal digits ([0-7])
\xnn hexadecimal bit pattern, where nn is 1-2 hexadecimal digits ([0-9a-fA-F])
\unnnn Unicode character, where nnnn is exactly 4 hexadecimal digits ([0-9a-fA-F])
\u{nnnn ...} Unicode character(s), where each nnnn is 1-6 hexadecimal digits ([0-9a-fA-F])
\cx or \C-x control character, where x is an ASCII printable character