Capturing the camera on android in the background with preview after processing with OpenCV

So I solved my own problem. If the app is in the foreground I need to show the preview, which I can most easily do with opencv, but then in the background I just want to process the frames which opencv can't do so I need to use CameraX. So why not do both.

I used a singleton usecase to keep track of whether the app is in foreground or background:

object AppStateUseCase {
    enum class AppVisibilityState {
        BACKGROUND,
        FOREGROUND,
    }

    private var _appVisibilityState = MutableStateFlow(AppVisibilityState.FOREGROUND)
    val appVisibilityState = _appVisibilityState.asStateFlow()

    fun updateAppVisibilityState(newState: AppVisibilityState) {
        _appVisibilityState.value = newState
    }
}

So I spawn the camera preview with this code which gets called from the main activity:

fun cameraPreview(
    context: Context,
    lifecycleScope: CoroutineScope
): JavaCamera2View {
    val view = JavaCamera2View(context, -1)
    lifecycleScope.launch { visibilityStateListener(view) }
    view.visibility = SurfaceView.VISIBLE
    if (!hasRequiredPermissions(context, REQUIRED_PERMISSIONS)) {
        throw IllegalStateException("startPreview camera called without necessary permissions")
    }
    view.setCameraPermissionGranted()
    view.setCvCameraViewListener(OpencvDetector())
    return view
}

private suspend fun visibilityStateListener(view: JavaCamera2View) {
    AppStateUseCase.appVisibilityState.collect {
        when (it) {
            AppStateUseCase.AppVisibilityState.BACKGROUND -> view.disableView()
            AppStateUseCase.AppVisibilityState.FOREGROUND -> view.enableView()
        }
    }
}

Then in the main activity the onStart and onStop update the state:

override fun onStart() {
    AppStateUseCase.updateAppVisibilityState(
        AppStateUseCase.AppVisibilityState.FOREGROUND
    )
    super.onStart()
}

override fun onStop() {
    AppStateUseCase.updateAppVisibilityState(
        AppStateUseCase.AppVisibilityState.BACKGROUND
    )
    super.onStop()
}

Now I don't bind to the service, I just use LifecycleService and manage CameraX with the IMAGE_ANALYSIS use case. The notable part is in analyzeInBackground which skips analysis when the app is in foreground because the main activity handles that. It might be worth to consider that by default opencv and camerax will give frames in different resolutions. CameraX defintily has a way to do that so it's worth checking out:

class CameraService: LifecycleService() {
    private lateinit var cameraProviderFuture : ListenableFuture<ProcessCameraProvider>

    private fun analyzeInBackground(imageProxy: ImageProxy) {
        if (AppStateUseCase.appVisibilityState.value
            == AppStateUseCase.AppVisibilityState.FOREGROUND) {
            imageProxy.close()
            return
        }
        
        Detector.analyze(imageProxy)
        imageProxy.close()
    }

    override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
        when (intent?.action) {
            Actions.START.toString() -> startService()
            Actions.STOP.toString() -> stopSelf()
        }

        return super.onStartCommand(intent, flags, startId)
    }

    private fun startService() {
        val notification = NotificationCompat.Builder(
            this,
            EmpressApp.NotificationChannels.PROCESSING_CAMERA.toString()
        )
            .setSmallIcon(R.drawable.ic_launcher_foreground)
            .setContentTitle("Processing the camera feed")
            .setContentText("")
            .build()

        cameraProviderFuture = ProcessCameraProvider.getInstance(this)
        cameraProviderFuture.addListener(Runnable {
            val cameraProvider = cameraProviderFuture.get()
            bindAnalyzer(cameraProvider)
        }, ContextCompat.getMainExecutor(this))

        startForeground(1, notification)
    }

    enum class Actions {
        START,
        STOP,
    }

    private fun bindAnalyzer(cameraProvider : ProcessCameraProvider) {
        val cameraSelector : CameraSelector = CameraSelector.Builder()
            .requireLensFacing(CameraSelector.LENS_FACING_BACK)
            .build()

        val imageAnalysis = ImageAnalysis.Builder()
            .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
            .build()

        imageAnalysis.setAnalyzer(
            ContextCompat.getMainExecutor(this)
        ) { imageProxy ->
            analyzeInBackground(imageProxy)
        }

        cameraProvider.bindToLifecycle(this , cameraSelector, imageAnalysis)
    }
}

Now the last important thing left is to convert from ImageProxy to Mat when analzying in the background:

         private fun imageProxyToMat(imageProxy: ImageProxy): Mat {
            if (imageProxy.format != PixelFormat.RGBA_8888) {
                throw IllegalArgumentException("image proxy has the wrong format")
            }

            val mat = Mat(imageProxy.height, imageProxy.width, CvType.CV_8UC4)
            Utils.bitmapToMat(imageProxy.toBitmap(), mat)
            Imgproc.cvtColor(mat, mat, Imgproc.COLOR_RGBA2BGR)
            Core.rotate(mat, mat, Core.ROTATE_90_CLOCKWISE)
            return mat
        }

And here is the Detector and OpencvDetector class which were used through out the code for the sake of completeness:

class OpencvDetector: CvCameraViewListener2 {
    override fun onCameraViewStopped() {}

    override fun onCameraViewStarted(width: Int, height: Int) {}

    override fun onCameraFrame(inputFrame: CameraBridgeViewBase.CvCameraViewFrame): Mat {
        return Detector.analyze(inputFrame.rgba())
    }
}

class Detector {
    companion object {
        fun analyze(input: Mat): Mat {
            Log.i("foo", "analyzing")
            return input
        }

        fun analyze(input: ImageProxy): Mat {
            return analyze(imageProxyToMat(input))
        }
    }
}

I may have explained it a little too much in depth than needed but I was excited that I found the solution and also wanted to share if anyone finds it useful.

Update

I notice that sometimes my approach is unstable this happens because when there is an overlap between both camera api's running causes it all to crash. This can be fixed here's how I did it: I manage the cameraPreview through the cameraService and basically ensure that switching between the camera api's happens in correct order and with no overlap. Then I access the preview from the main activity by bounding the service. Here's a snippet that's responsible for core behavior

private suspend fun cameraBackendSwitcher(cameraProvider: ProcessCameraProvider, cameraSelector: CameraSelector, useCase: UseCase) {
        AppStateUseCase.appVisibilityState.collect {
            when (it) {
                AppStateUseCase.AppVisibilityState.BACKGROUND
                    -> {
                    cameraPreview.disableView()
                    cameraProvider.bindToLifecycle(
                        this@CameraService,
                        cameraSelector,
                        useCase
                    )
                }

                AppStateUseCase.AppVisibilityState.FOREGROUND
                    -> {
                    cameraProvider.unbind(useCase)
                    cameraPreview.enableView()
                }
            }
        }
    }

I then start this function at the end of bindAnalyzer, this required other refactors but this is the most important part so I'm showing it here.