Well, knowing that someone is annoying doesn't make him any less annoying. Maybe you could cope with it more easily in the beginning, but you still get pissed off during the years, especiallyif you have to work with him almost every day.
And I'm not really talking about Yoshiki as a mean dictator, I believe it was good for the band (at least I love everything they've done) but if everything has to be run through one person and he has the last word in everything and is a bit of a tricky person it gets REALLY frustrating at certain situations in a band.
I don't believe either that the musical issues were the only reason for Taiji leaving the band, but nothing more has been told to us. And if Taiji would've answered "no" when Yoshiki asked him to leave, I don't think he would've lasted long in the band. Or maybe there would still be X, who knows?