Should i buy more of each species to decrease agression?

My two female swords fight and my two dwarf gourami's fight, would having one or two more of each species diffuse this? they're in a 20gal with a betta btw. the betta is well behaved.