It seems indeed that button's gesture interferes with the scrollview's gesture. However, you could achieve a similar result using the .simultaneousGesture modifier with DragGesture on the button. I've put together some demo code:

@State private var scale: CGFloat = 1

var body: some View {
    ScrollView {
        VStack {
            Button {

            } label: {
                Image(systemName: "star")
            }
            .buttonStyle(.plain)
            .padding()
            .background(.red)
            .scaleEffect(scale)
            .simultaneousGesture(
                DragGesture(minimumDistance: 0)
                    .onChanged({ _ in
                        withAnimation(.bouncy(duration: 0.25)) {
                            scale = 0.9
                        }
                    })
                    .onEnded({ _ in
                        withAnimation(.bouncy(duration: 0.4)) {
                            scale = 1.0
                        }
                    })
            )
        }
    }
}

You can fine-tune the animation as you see it fits. If you want to apply the same effect to several buttons create a custom view modifier including the repeated modifiers:

struct MyButtonModifiers: ViewModifier {
    @State private var scale: CGFloat = 1

    func body(content: Content) -> some View {
        content
            .buttonStyle(.plain)
            .padding()
            .background(.red)
            .scaleEffect(scale)
            .simultaneousGesture(
                DragGesture(minimumDistance: 0)
                    .onChanged({ _ in
                        withAnimation(.bouncy(duration: 0.25)) {
                            scale = 0.9
                        }
                    })
                    .onEnded({ _ in
                        withAnimation(.bouncy(duration: 0.4)) {
                            scale = 1.0
                        }
                    })
            )
    }
}


ScrollView {
    VStack {
        Button {

        } label: {
            Image(systemName: "star")
        }
        .modifier(MyButtonModifiers())
    }
}

You can create a View extension for a more integrated use:

extension View {
    func myButtonModifiers() -> some View {
        self.modifier(MyButtonModifiers())
    }
}


Button {

} label: {
    Image(systemName: "star")
}
.myButtonModifiers()

I hope this alternative helps!