Do I need to learn system programming (e.g. WinAPI and Linux API) to be really good in reverse engineering? Or should I treat it as a reference so that whenever I encounter a system call, I go learn about it?
I want to be a reverse engineer as a part of my penetration testing "hobby" project and therefore I need advice about this issue.
Generally a reverse engineer translates assembly code to a higher level language, for that reason it is quite a necessity to be familiar with that language (or at least a similar language), usually understanding the intrinsics of those languages and how common language constructs are translated to the lower level is very helpful.
Understanding system level APIs is required when you try to understand a programs execution (such as network interfaces, file operations, UI, e.g. read(), recv(), etc) but is not required when you're reverse engineering a contained piece of code, i.e. it doesn't read files, send/receive packets. Multiprocessing/threading might complicate things for you if you're uneducated about those, to name one example.
Another topic you should be familiar with if you're interested in reverse engineering and didn't mention is knowing assembly languages for the architectures you'll be reverse engineering (PCs are usually Intel's x86 and AMD's x86_64). There are instruction set manuals for both, which cover the assembly languages in depth and provide both a reverence manual and introduction to general approaches and concepts (such as registers, the stack, etc.).
You can obviously learn as you go, and that's a valid approach towards learning reverse engineering, for that you should really learn by examples.
you could head over to crackmes.de or http://www.tuts4you.com and start on the beginner reverse-engineering binaries and read as many write-ups (solutions) as you can!
Another good novice resource is lena141's series of dynamic RE videos.
Generally system programming is not required, but recommended...
It is also important to understand the application flow, the CPU architecture (stack, heap, exceptions), file format (PE, ELF).
I suggest two good resource:
Crackmes.de
Reverse engineering for beginners - https://beginners.re/RE4B-EN.pdf
You need to know the basic general principles behind system programming for each OS you're interested in - e.g. what a system call is and, say, maybe how virtual memory is managed. With time you will identify areas of system programming you are more interested in for your day to day work.
APIs - treat them as reference to look up specifics of each call.
The only way to learn reversing is to reverse a lot. Studying encyclopaedias is a sure way of procrastination :)
