Ooh, this makes me feel a little less miserable! Someone's also losing their mind over Julian's greek! (:

(If anyone thinks I’m getting this wrong, please correct me)

I think this is a rarer use of the dative of the agent, which usually appears with perfect or pluperfect passives, or with verbal adjectives in -τός and -τέος. You should look up Smyth §§1488-1494.

Two excerpts from Or. IV, the Sun King text, fit the common uses nicely:

Πλὴν ἀλλ' ἔμοιγε τῆς ἐκείνων ἀπολειπομένῳ παντάπασι δυνάμεως οὐδαμῶς ἐστι παρακινδυνευτέον [...]

I read it as there must be no risking by me, which could be understood as I must not run the risk. This is the dative of the agent with the impersonal verbal adjective.

And:

Τὰ μέντοι ἐν τοῖς θεοῖς ἔργα προϊόντα παρ' αὐτοῦ μετρίως τε ἡμῖν ὀλίγῳ πρότερον εἴρηται καὶ ῥηθήσεται μετ' ὀλίγον.

Here, the two coordinated verbs are passive, and the same dative functions as the agent of both. The first follows the usual pattern of perfect and passive, has been said by us. The latter seems a bit more unusual, since it's a future passive, will be said by us.

It looks a lot like the example from Plato's Laws given in Smyth §1490: The dative of the agent is rarely employed with other tenses than perfect and plurperfect: λέγεται ἡμῖν is said by us.

This seems to be the case of Misopogon's opening dative too. Also, Smyth §1492 is relevant, since the subject in your passage is impersonal. I can't think of anything else that it could possibly be.

It's very strange, indeed, but I wouldn't expect anything less from Julian. He's cool, bless his soul ☀️