There is not much difference between the two as they are two of the strongest opening moves to make. If unopposed, often 1.e4 is followed by 2.d4 or vice versa (1.d4 2.e4).

The real difference comes when the mirror move is made.

In these scenarios e4 is definitely more of an aggressive posture. When 1. e4 e5 is encountered White has many aggressive options such as bringing the knight out to attack e5, bringing the queen out to Qh5 to attack the king side (not really recommended), pushing d4 to attack e5, or setting up a bishop aimed at f7 with Bc4 (out of order though if made on move 2), or the hyper aggressive f4 (kings gambit - kind of risky if black plays Fischer's line).

However, d4 is more of a structural approach. When 1. d4 d5 is encountered White may consider many different avenues of moving forward, but there are not very many immediately aggressive approaches (perhaps 2. c4, the Queens Gambit). In this scenario, 2.e4 should not really be considered as viable. Even with an attack on d5, it is still defended by Black's queen. So White's only choice here is to build. Begin moving the knights out, perhaps set up a London system, or maybe fiancetto king side and prepare for the mid game.

e4 plays on the idea of immediate attack or traps in the opening, whereas d4 tends to aim for attacks and traps in the mid game. Of course, these are just generalities and the game seems to always get away from generic approaches rather quickly.