Problem
Currently, Octave is not fully compatible with the new "Windows Terminal App" (which become the default console in Windows 11).
During Octave installation you should see the following warning:
Octave doesn't work correctly when the default console on Windows is set to the new Terminal app. Check if the default is set to the Windows Console Host and show advice if it is not.
Consequences
This unsupported configuration could lead to unexpected behavior of the Octave CLI like the "weird" characters and colors mentioned in your question.
Also, you can have both CLI and command window opened simultaneously and displaying exactly the same content (you could also observe strange text overlap and difficulty to scroll-up/down).
Workaround
As stated in comment, a workaround consists of setting the "Windows Console Host" (i.e. the old cmd.exe console) as the default windows console (see screenshot of the settings screen below) [from here].
A way to access this setting is to type "terminal" in the Windows search box and select "Choose a terminal host app for interactive command-line tools".

Potential solution to come
A "new cross-platform command window widget" is currently in development and may be able to solve this issue.