How to implement mutual calling between different programming language by shared library?

Java is compiled to bytecode and interpreted on JVM, which is a separate process, how does JNI execute an external C shared library on it?

The JVM (in this case the Java bytecode interpreter in it) is just a normal native program for whatever platform you are on. It takes Bytecode, looks at each instruction, and does whatever the instruction tells it to do.

As such, it calls functions in a shared library exactly like any other program on your computer would. It loads the library, and uses system-provided functions to look up the function by name. This gives it a pointer to the start of the function, which it can use to jump to that instruction and call it.

The compiler that generates the bytecode takes Java programs in text form and generates a list of bytecode instructions. If you tell it to call a function in a native library, it needs a description of how to call that function (at the least what parameter types it takes, and its return type), this is what the native line in Java code is for. Then it just writes out an instruction that says "load this library (if needed) and call the function with that name in it".

Most of this can not be done in a language like C or Java directly, but is fairly easy to do in machine language. So if you want to know how this could work, I recommend you learn some assembler. 68000 assembler is really clean and nice to understand, but any assembler will do. You don't need to learn it deeply. Learning how to write a recursive function in assembler will teach you everything you need to know.

Alternately, there are libraries (like libffi) that actually contain the required machine code for calling a function whose parameter types and return type you don't know until runtime, and wrap it in a nice C API. That assembly language code won't mean much to you, but may give you an idea.

If you're curious on how a compiler and a bytecode interpreter (like the ones in Java) work, I did a video tutorial on making a programming language once. But while it's all written in C++ (which is quite similar to Java), why I do some things may not really be clear until you learn the basics of assembler.

As to what JNI is: JNI is (at its root) just the native code that the bytecode interpreter uses under the hood to do its work. To e.g. create objects, look up methods in them etc. Every bytecode instruction basically calls a JNI function under the hood. JNI is just there so that native code can tell the Java bytecode interpreter what it is doing, as Java does not "see" the native variables in a native library that point to Java objects, for example, so the garbage collector can't know that the native code in the library still needs that object. So JNI for example supplies some functions that tell the garbage collector "this object is still needed, don't collect it yet, even if no bytecode references it anymore".