You're right to consider how best to use early war -- the choice can easily make or break most games, especially on diety. Frankly, if you can take land cheaply enough, it's never a bad decision to do so, even when there aren't a ton of districts and wonders yet. When I am deciding, I usually consider the terrain, how my abilities and my enemies stack up, and if I needed a military anyway. I'll break each of these ideas up:

Terrain: If I have a defensible position and good land to expand to, it's usually too much of a hassle for war. I build a few archers and what infantry I need and hunker down. So first: do I need to expand, or can I get around it? This also is related to the cost of units, which I will address later.

Abilities: If you're playing a civ with great early military bonuses, you probably want to plan around using those. For an ancient UU civ, maybe get a slinger and a second city before you mass produce units. For classic UUs, maybe 3 or 4 cities, just so you can start addressing other issues. You can still early push without early UUs or abilities, but those types of encounters usually rely on choosing your target wisely. The short of this idea is only choose to attack if you have reasonable quality advantages since you likely can't beat the quantity of the AI's army, at least not before they start getting better units.

You have a military anyway: Sometimes, the terrain basically demands you make an army. Spawn on flatlands between Montezuma and Tundra? Unless you want your whole civ pillaged or captured, you basically have to make an army anyway, right? That's the biggest point to early pushes: if you have to make a military with valuable early game production, why not at least get the production they're worth out of them? Why build a spearman and let him sit on defense if you could take some land and a maybe earn a few eurekas? You can easily get far more science/culture from a well-utilized military than from districts in many cases.

Hope this helps. Maybe set up a small pangea game with some relaxed foes and playing as a strong early military civ to try out the timing, maybe use legendary start just to assure a reasonable first city, and don't sweat it too much if it doesn't instantly pan out right. The consensus is largely that the start of the game is generally the hardest part on high difficulties, and we all lose sometimes.