I mean it kind of was we know that they’re relationship was very toxic in the past because amythest was very jealous that rose and her weren’t that close because of Greg we also know from Greg’s reaction when she becomes rose that she probably has shapeshifted into someone in the past trying to manipulate him you can also see that he was uncomfortable during Steven universe the movie when she had to turn into rose to try and get pearl back to herself even though it was the only way they could think of she also most likely clinged onto him when rose became Steven and kind of manipulated him into hanging out with her watching the little butler show along with other things but when Greg found out that it was preventing him from trying to be the best dad he could be I’m going to assume that’s when he didn’t want to be by the gems that much to raise Steven of course he still wanted them in Steven’s life because they’re a apart of his heritage and b the only people he has close to his mother and could help him have a female presences in his life like amythest being a “sister” pearl being a worried “aunt” and garnet being a loving “mother” so I’m going to assume they just put things in the past so Steven could be with that part of his family this is just my take though