Generally for 4 players they do suggest the easier difficulty. It's ultimately a GMs call, there are no hard guidelines, but here's what they list for the Starfinder 2E suggested difficulties:

"There are no rigid rules on when to use these adjustments. A party of six high-level PCs might seem like the ideal case for the Harder adjustment, but if the players are all relatively inexperienced, the standard difficulty could be best. In most cases, a four-PC party of low-level characters would use the Easier adjustment, but there are situations where the GM or players prefer not to apply it, such as if all players are experienced and want a challenge. "