Orgies are too complicated. Too many people feel left out of the fun if they are not getting fucked. Spouse Swapping is easier with only 4 people needed to reach agreement on pre conditions.
No single men allowed. (Unless you're holding a gangbang or a gay orgy.) Thirsty dudes will overwhelm your orgy. In general, you need to be a hetero couple, lesbian couple, or single woman to get into the swing scene. If you're a gay man, and you have trouble finding gay orgies, there is probably something socially or physically bad wrong with you.
It's a stereotype, but gay guys know how to party and don't fuck around about fucking around.
See if there is a local swingers group on FetLife.
The best orgies just kind of spontaneously happen. You usually have to have a close knit core of friends. After parties for kink events and such.
Sharing Ecstasy/MDMA with close friends can make one happen.