Hoo boy. Yeah, finding a group is a very difficult task, and nearly as difficult (if not more so) than convincing family. Lucky for me my current F-in-law is at least partially on board and has seen the extent I've already gone through to prepare for the family for hard times. I helped them get their cabin on the Rim stocked and though Mom-in-law is Aunt Esther to my Fred Sanford she's a product of depression era parents herself. [Side note: I think her biggest gripe with me is that their daughter is 20+ years younger than me.]
As for groups, different ideologies, religious beliefs, race and political views can have a major negative effect on cohesiveness if you have many polar opposites. That will often lead to dissent among the group. I personally don't care for and won't tolerate bigotry among my teams at work, and won't do so for a survival group either. I don't care if you're a black lesbian puerto-rican jew or an asian muslim communist. If you have the right attitude and skill set you're welcome to apply.
(But if you're more concerned with those values than the group, you're welcome to leave to)
Some groups I've encountered in my lifetime have had extremist bents, whether it be religious, racial or politically motivated. I've seen *********** christian zealots and hippie commune types that just want to live "green" and feed the world and abhor violence. I've also seen just about everything in between.
As has been stated already, leadership is an issue and so is individual purchases of group property. If one family leaves the group, not only do others tend to take sides, but there is also the risk of non-preppers purchasing the property. There is only so much that can be done to prevent someone from making the purchase pre-SHTF. Having and maintaining a cohesive and cooperative group outside of a family unit can take years, and even then there is risk.
I touched on individual values shortly. I'll give another example now. When I was in the Legion we were forced in a way to put our individual bias behind us or lock it away for the common good. That included both teaching us the history and honor of the Legion, as well as having it beaten into us. When you sign up you are theirs for the contract duration (5 years minimum) and there is no one to complain to, no congressman to write to, if your corporal or sergeant beats you to a pulp because he doesn't like the way your boots are polished or the line of your shoulders as you stand at attention. It is harsh in a sense that makes R. Lee Earmy in Full Metal Jacket look like a pansy teaching flower arrangements. My point in stating that is that we had people from all over the globe in my units. We had Croats and Serbs, Arabs and Israelis, Brits and Germans, Asians and Africans. Whatever our religious or political beliefs going in, we came out as Legionnaires and dedicated to each other first and our religious/political beliefs second. I had no worries that if we went into Eastern Europe that the Serb and Croat members of the unit would turn on each other, nor did I worry in Africa that the Tutsi and Hutu members would turn on each other. We were of the Legion, we were Legionnaires and therefore we were family. I've seen similar cohesive relationships develop among various other militaries as well over the globe.
My point in that is a a veteran you might find better luck with other former military people. Additionally, be careful of compromise if the group is too extremist for your taste. In small scale natural disasters people have come together for mutual aid and have crossed political, racial, religious and economic boundaries to do so. That mutual aid, however, is quickly lost after the event is over, so don't count on it for long term either.
The online story/novel "Lights Out" gives a small overview on various different survival groups and the choices one man and his family make as to which type of group to join with during an EMP event. Overall it's a good story, but please understand that the main characters get unseasonably lucky in several events.
I have a few people whom I work with on a regular basis that I would gladly have with me in an SHTF situation. Some are retired and live near my properties, while others are hired hands. Not all know all of my preps but they know that I am prepared. Some are sure things for a major event while others I'm still feeling out before providing further information.
I wish you luck in your search and hope you can find or create a group that suits you.