SwiftUI Sheet with ScrollView like in Apple Maps

I want to create a sheet in SwiftUI, it has 3 PresentationDetends Types [.fraction(0.2), .medium, .large]. When I add these 3 types, I can move the Sheet to the 3 sizes I want.

However, I also want to display a large Text on the sheet, and when I do it with a ScrollView, I can no longer move the Sheet downwards because I always trigger the ScrollView. I tried recreating the ScrollView using ScrollViewUI from UIKit, I then blocked the scroll when the contentOffset was < 0, that worked a bit, because then I was no longer able to scroll down on the ScrollView but I could close the Sheet as intended. Then I tried using selection: PresentationDetend on the sheet with the .PresentationDetends modifier.

That worked, however, when I set the variable to change to .fraction(0.2) it does change to the specified size, but then the Sheet shortly jumps up and down, and then it just resizes (it is weird to explain, I think this is some visual bug or so idk). Do you guys have any Idea how I can achieve a ScrollView with large Text Content on a Sheet and still being able to close the sheet when dragging down (Dragging down when the scrollview content is on top, like on Apple Maps).

I tried different thing, but right now, I was not able to get this type of functionality running.

Update

Durul Dalkanat provided an excellent answer to my Question that works quite nice. However, I forgot to mention that I am using a TabView in my sheet (I didn't thought this would make any difference in functionality, next time I do better), and it seems to be the case that the TabView is the part that is blocking my movement, not the ScrollView in the Sheet.

This is the Example Code:

            .sheet(isPresented: $isSheetPresented) {
                    TabView {
                        
                        ForEach(0..<5, id: \.self) { i in
                            
                             ScrollView {
    
                             //Content
    
                             }
                            
                        }
                        
                    }          

.tabViewStyle(PageTabViewStyle(indexDisplayMode: .always))
 }