In my understanding the green stuff (my addition) would be enough?
Yes and no. Networks work bidirectionally, so for a full-duplex network you might need to switch one direction independent of the other, so you'd need the full matrix.
This is rather academical though. In a circuit-switched network you need a crossbar switch with only the green part of the grid, for both directions 'simultaneously'. Regardless of duplex mode, in a switched circuit two nodes A & B talk only to each other, not A to B and B to C.
In a packet-switched network there's no crossbar switch, at least not a physical one. Packets are routed across a backplane that provides sufficient bandwidth for all potential traffic (non-blocking) or not (blocking architecture).
I guess AST wanted to point out the logical difference between a 1:N and an N:N communications network, rather than show how (repeater) hubs and switches are really built.