SwiftUI: PhotosPicker fails to load Transferable image

The following code seems to work great in a new app, but in my existing app it always fails with the error message below. I struggle to find anything in my app that causes this issue.

Here's the code:

@State private var isShowingPhotoPicker = false
@State private var selectedItem: PhotosPickerItem? = nil
 Button("Select image") {
      isShowingPhotoPicker.toggle()
}
.photosPicker(isPresented: $isShowingPhotoPicker, selection: $selectedItem, matching: .images, photoLibrary: .shared())
.onChange(of: selectedItem) { newItem in
                if let newItem {
                    Task {
                        do {
                            if let image = try await newItem.loadTransferable(type: NSImage.self) {
                               //Do something with the image....
                            }
                        } catch {
                            NSLog("\(error)")
                        }
                    }
                }
            }

It fails on both, macOS and iOS. The error in catch is

importNotSupported

and I also get this long warning and error:

Upload preparation for claim 5C3B57B9-94C6-4CC6-A632-C473FE8FB85A completed with error: Error Domain=NSCocoaErrorDomain Code=513 "“uuid=59655946-D65D-418D-868E-F62EE10EB667&library=1&type=1&mode=1&loc=true&cap=true.jpeg” couldn’t be copied because you don’t have permission to access “CoordinatedZipFilep40sph”." UserInfo={NSSourceFilePathErrorKey=/Users/daniel/Library/Group Containers/group.com.apple.Photos.PhotosFileProvider/File Provider Storage/photospicker/uuid=59655946-D65D-418D-868E-F62EE10EB667&library=1&type=1&mode=1&loc=true&cap=true.jpeg, NSUserStringVariant=( Copy ), NSDestinationFilePath=/var/folders/26/cfq1mmmn7wl1nbqwjlxf4zw00000gt/T/CoordinatedZipFilep40sph/uuid=59655946-D65D-418D-868E-F62EE10EB667&library=1&type=1&mode=1&loc=true&cap=true.jpeg, NSFilePath=/Users/daniel/Library/Group Containers/group.com.apple.Photos.PhotosFileProvider/File Provider Storage/photospicker/uuid=59655946-D65D-418D-868E-F62EE10EB667&library=1&type=1&mode=1&loc=true&cap=true.jpeg, NSUnderlyingError=0x600003e60b10 {Error Domain=NSPOSIXErrorDomain Code=1 "Operation not permitted"}} Claim 5C3B57B9-94C6-4CC6-A632-C473FE8FB85A failed during preparing for uploading due to error: Error Domain=NSCocoaErrorDomain Code=513 "“uuid=59655946-D65D-418D-868E-F62EE10EB667&library=1&type=1&mode=1&loc=true&cap=true.jpeg” couldn’t be copied because you don’t have permission to access “CoordinatedZipFilep40sph”." UserInfo={NSSourceFilePathErrorKey=/Users/daniel/Library/Group Containers/group.com.apple.Photos.PhotosFileProvider/File Provider Storage/photospicker/uuid=59655946-D65D-418D-868E-F62EE10EB667&library=1&type=1&mode=1&loc=true&cap=true.jpeg, NSUserStringVariant=( Copy ), NSDestinationFilePath=/var/folders/26/cfq1mmmn7wl1nbqwjlxf4zw00000gt/T/CoordinatedZipFilep40sph/uuid=59655946-D65D-418D-868E-F62EE10EB667&library=1&type=1&mode=1&loc=true&cap=true.jpeg, NSFilePath=/Users/daniel/Library/Group Containers/group.com.apple.Photos.PhotosFileProvider/File Provider Storage/photospicker/uuid=59655946-D65D-418D-868E-F62EE10EB667&library=1&type=1&mode=1&loc=true&cap=true.jpeg, NSUnderlyingError=0x600003e60b10 {Error Domain=NSPOSIXErrorDomain Code=1 "Operation not permitted"}} Error loading public.image: Error Domain=NSItemProviderErrorDomain Code=-1000 "Cannot load representation of type public.jpeg" UserInfo={NSLocalizedDescription=Cannot load representation of type public.jpeg, NSUnderlyingError=0x600003e64390 {Error Domain=NSCocoaErrorDomain Code=513 "“uuid=59655946-D65D-418D-868E-F62EE10EB667&library=1&type=1&mode=1&loc=true&cap=true.jpeg” couldn’t be copied because you don’t have permission to access “CoordinatedZipFilep40sph”." UserInfo={NSSourceFilePathErrorKey=/Users/daniel/Library/Group Containers/group.com.apple.Photos.PhotosFileProvider/File Provider Storage/photospicker/uuid=59655946-D65D-418D-868E-F62EE10EB667&library=1&type=1&mode=1&loc=true&cap=true.jpeg, NSUserStringVariant=( Copy ), NSDestinationFilePath=/var/folders/26/cfq1mmmn7wl1nbqwjlxf4zw00000gt/T/CoordinatedZipFilep40sph/uuid=59655946-D65D-418D-868E-F62EE10EB667&library=1&type=1&mode=1&loc=true&cap=true.jpeg, NSFilePath=/Users/daniel/Library/Group Containers/group.com.apple.Photos.PhotosFileProvider/File Provider Storage/photospicker/uuid=59655946-D65D-418D-868E-F62EE10EB667&library=1&type=1&mode=1&loc=true&cap=true.jpeg, NSUnderlyingError=0x600003e60b10 {Error Domain=NSPOSIXErrorDomain Code=1 "Operation not permitted"}}}}

My app is not sandboxed on macOS. What am I missing?

EDIT: Now things get weird, I have added a GIF and PNG to my photos library and the PhotosPicker has no problems getting that. But all other formats fail. JPEG and HEIC fails.