How to properly handle negative values when calculating the difference between two images in OpenGL?

Question

I am working on a project using OpenGL, where the server needs to render two images: a high-quality image and a low-quality image. The goal is to compute the difference image between the two and send this difference to the client. The client then renders the low-quality image and combines it with the received difference image to reconstruct the high-quality image.

On the server side, I am calculating the difference between the two images in the fragment shader by directly subtracting them and outputting the result as a texture.

#version 330 core
out vec4 FragColor;

in vec2 TexCoord;

uniform sampler2D HighTexture;
uniform sampler2D LowTexture;

void main()
{
    FragColor = texture(HighTexture, TexCoord) - texture(LowTexture, TexCoord);
}

then client side:

#version 330 core
out vec4 FragColor;

in vec2 TexCoord;

uniform sampler2D LowTexture;
uniform sampler2D DiffTexture;

void main()
{
    FragColor = texture(LowTexture, TexCoord) + texture(DiffTexture, TexCoord);
}

However, I am facing an issue when the result of the subtraction is negative(for example: high image has shadow): OpenGL automatically truncates the negative value to zero, which leads to incorrect rendering on the client side.

Both images are of type RGB, and the texture format is GL_UNSIGNED_BYTE.

If I switch to using a floating-point texture format such as GL_RGBA32F, the size of the difference image becomes much larger, which goes against the goal of reducing the bit rate during transmission. Since the main purpose of sending the difference image is to reduce the amount of data transferred, using a floating-point texture would significantly increase the data size and nullify the advantage of compression.

What I need help with:

How can I handle negative values while still keeping the texture size small enough for efficient transmission?
How can I prevent OpenGL from automatically truncating negative values when calculating the difference between the two images?
Is there a way to preserve negative values in the subtraction result, or should I change the texture format to something that supports floating point values?

A texure format can never be GL_UNSIGNED_BYTE, at best, that's the format of the data you are uploading to the texture. What is the internal format of the texture you create? GL_RGB8? Since you only deal with integer values in first place, why not render to a integer texture at all? There are 8 and 16 bit integer formats that both support negative numbers. — BDL
– BDL, Commented Dec 6, 2024 at 15:55
Do the computation in a larger signed type on the GPU, add 128 and clip to the [0,255] range before casting back to unsigned byte. — Cris Luengo
– Cris Luengo, Commented Dec 6, 2024 at 19:37

Yakov Galka · Accepted Answer · 2024-12-09 05:55:33Z

2

The difference between two real numbers in the [0, 1] range would be in the [-1, 1] range, i.e. twice as big. So if you want to stay in the realm of real numbers, you'd need to change your difference texture to use GL_RGBA8_SNORM internal format. However that will be a lossy conversion, because you'll be encoding a range twice as big in the same eight bits.

If you truly need a lossless conversion, then you need to work in modular arithmetic. Switch all textures (high, low, and diff) to use GL_RGBA8UI -- so the arithmetic will wrap-around instead of truncating. EDIT: You'll also need to use usampler2D instead of sampler2D in the shader, and uvec4 instead of vec4 for the FragColor.

edited Dec 9, 2024 at 5:55

answered Dec 6, 2024 at 16:01

Yakov Galka

72.8k16 gold badges148 silver badges226 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

A_____ A_______ · Accepted Answer · 2024-12-06 19:26:49Z

on server you can do:

void main()
{
    FragColor = texture(LowTexture, TexCoord) - texture(HighTexture, TexCoord) + vec4(0.5, 0.5, 0.5, 1.0);
}

and on client side:

void main()
{
    FragColor = texture(LowTexture, TexCoord) - texture(DiffTexture, TexCoord) - vec4(0.5, 0.5, 0.5, 1.0);
}

for example if highTex have 16x16 pixels per one lowTex pixel and lowTex pixel is avg of highTex pixels then diffTexture pixels will be in bounds from 0 to 1 in ~98.75% (with evenly distributed values on real images result can be a little different but not too much) of cases with this method
here is small js code which tests it for 50_000 * 16*16 = 38_400_000 rgb pixels of highTex

const testIterCount = 50_000*3
const result = {error: 0, ok: 0}
for (let i = 0; i < testIterCount; i++) {
    const high = new Float32Array(16*16).map(a=>Math.random())
    const low = high.reduce((a,b)=>a+b)/high.length
    const testResult = high.reduce((counter,highVal)=>{
        const serverVal = low - highVal + 0.5;
        if (serverVal >= 0 && serverVal <= 1) {
            counter.ok++;
        } else {
            counter.error++;
        }
        return counter
    },{error: 0, ok: 0})

    result.error += testResult.error
    result.ok += testResult.ok
}
console.log(`ok ${ (result.ok/(result.ok+result.error)*100).toFixed(2) }%; error ${ (result.error/(result.ok+result.error)*100).toFixed(2) }%; `)

but also if you want to reduce bytes count you send to client you can just use jpeg or webp and just send low quality image and high to user or you are using compression method witch compress raw DiffTexture better than jpeg or webp do?

Collectives™ on Stack Overflow

How to properly handle negative values when calculating the difference between two images in OpenGL?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related