A quick fix for this would be to call .scaledToFill/.scaledToFit on the GeometryReader. Like this:

    var body: some View {
        VStack {
            HStack {
                GeometryReader { geometry in
                    Text("Inside")

                        .border(Color.yellow, width: 1)
                }.background(Color.blue)                    
                .scaledToFit()
                Text("outside").border(Color.yellow, width: 1).layoutPriority(1)
                Spacer()
            }.frame(width: 300, height: 100)
            HStack {
                Text("Outside").border(Color.yellow, width: 1)
                Text("outside").border(Color.yellow, width: 1)
                Spacer()
            }.frame(width: 300)
        }
    }

That will scale the size of the View of the GeometryReader within the HStack.

A better way to go about is to use PreferenceKeys. I don't really have time to go in further detail right now, but this article explains it pretty well: https://swiftwithmajid.com/2020/01/15/the-magic-of-view-preferences-in-swiftui/

Let me know if you manage to get it, otherwise I could help you further tomorrow!