It is likely the magnetometer needs to be calibrated before it can be used. Before calibration is applied to the magnetometer's output values, comparing any 2 of the 3 orthogonal sensors will be difficult.
Search and learn the terms hard iron and soft iron distortion / calibration with respect to magnetometers. Hard iron distortion is an offset error. Find the maximum (point a particular magnetometer toward magnetic north) and the minimum (point the same magnetometer away from magnetic north) values of one of the magnetometers. Calculate an offset value such that when applied the maximum and minimum magnitudes are the same. Soft iron distortion is a magnitude error. Find the maximum and minimum values for all 3 sensors. From this find the total range of all 3 sensors. Arbitrarily pick one of the sensors ranges and calculate magnitude adjustment values for the remaining sensor ranges such that after applying them all 3 sensors have the same range.
At this point let us narrow the scope of the electronic compass to simplify the project. Let us constrain the compass to only be used on a table which is parallel to the surface of the earth. At which point we only need to use 2 of the sensors (the two that lie in the plane of the table).
To calculate the angle the magnetometer is pointing with respect to magnetic north we take the arctangent of the ratio of the 2 magnetic sensor which lie in the plane of the table. However, this math function only gives us angles for half the desired angles. Fortunately, the C programming math library contains a special function atan2() which can return a full circle of angles (from here):
Synopsis. The atan2() function divides the first argument by the second and returns the arc tangent of the result, or arctan( y / x ) . The return value is given in radians, and is in the range -π ≤ atan2( y , x ) ≤ π.