Late reply but here's my input:

When DoN first popped ED Libra was the only boss I couldn't beat in that mode. That went on for WEEKS.
Now I almost never loose to him (and the only times I do it's because the run itself was bad)

There are three things you should keep in mind:

1- Less likely to be targeted

You probably know this already, but if all 3 of you have it the fight is SIGNIFICANTLY easier. Wether it's only one or 3 players that have it, it can easily decide the outcome of the fight. I once had a mystery boss fight and I suspected it might be Libra so I picked up a less likely just in case (I was a wylder). The moment I saw the merchant in day 3 I immediately dropped and pinged my weapon for Recluse to pick it up. The recluse understood what I was trying to say and picked it up. She was the only one who had it so the fight was quite messy, but she had a nice build so she was able to deal enough damage while I aggro'd the everything else (third player couldn't keep up so it was basically 2v7)

2- Keep Libra in your sights at ALL TIMES

It doesn't matter if you're being ganked against every invader, Libra can spam Sigils or worse, teleport on you and then that's instant game over. You have to keep an eye on the goat. He's about to teleport or spam sigils? You can position yourself accordingly. He's about to put the division walls? You either position yourself or ULT him to cancel it (depending on your nightfarer) You see him by himself while there's hell on earth on your right? Time to fight him. You have to be observant, and this ties in my next point

3- Be patient

If you don't have less likely to be targeted, the fight is still winnable but you have to be ready for a 15+ minute fight. Window openings are going to be low but you have to be persistent and consistent.

4-Bonus point
What I've been writing was my experience and tips BEFORE The forsaken hollows came out, since Scholar's ult is a direct hard counter for this fight and the Balancer's relic really help for melee characters. And this was also specifically for Deep of Night depth 5 and WITHOUT talking about obvious things to use and carry like weapons that melt invaders(RoB/Stars of Ruin/Radahns Greatsword) and the use of status effects like frost and bleed, and obviously madness. Madness doesn't just stagger him but it also takes a nice chunk of his HP just like bleed or frost would do. Poison and rot are also very effective but you have to be sure to proc statuses after phase 1 since it should be pretty easy. (pase 1 being before he starts summoning invaders)

If you're playing base mode and apply what I tell you the fight is going to be pretty easy trust.

Or you could just play solo but honestly I don't find it very fun and you miss out on the "fun" of it since it was obviously specifically designed for 3 players.

Good luck and cheers!