Based on a clue from gilles' answer, I had a process marked "Z" in top (<defunct> in ps) that was using system resources, it even had a port open that was LISTEN'ing and you could connect to that port. This was after executing a kill -9 on it. Its parent was "1" (i.e. init) so theoretically it should just be repeaed and disappear. But it wasn't, it was sticking around, though not running, and "not dying"
So in my case it was zombie but still consuming resources...FWIW.
And it was not killable by any number of kill -9's
And its parent was init but it wasn't being reaped (cleaned up). I.e. init had a zombie child.
And reboot was not necessary to fix the problem. Though a reboot "would have worked" around the problem/made it faster shutdown. Just not graceful, which was still possible.
And it was a LISTEN port owned by a zombie process (and a few other ports too like CLOSE_WAIT status connected localhost to localhost). And it still even accepted connections. Even as a zombie. I guess it hadn't gotten around to cleanup up the ports yet so incoming connections were still added to the tcp listening port's backlog, though they had no chance of being accepted.
Many of the above are stated as "impossible" on various places in the interwebs.
Turns out that I had an internal thread within it that was executing a "system call" (ioctl in this instance) that was taking a few hours to return (this was expected behavior). Apparently the system cannot kill the process "all the way" until it returns from the ioctl call, guess it enters kernel land. After a few hours it returned, things cleared up and the sockets were all automatically closed, etc. as expected. That's some languishing time on death row! The kernel was patiently waiting to kill it.
So to answer the OP, sometimes you have to wait. A long time. Then the kill will finally take.
Also check dmesg to see if there was a kernel panic (i.e. kernel bug).