Our guild revamped our recruiting policy about a month ago. Keeping the crazies out is a huge chunk of running a guild properly. So what worked?
First of all, we force people to create an account on our forums in order to put an application in. The instructions are very clear and easy to follow. This step probably cuts down on a lot of people who should not be in our guild. It means having a valid e-mail address and being able to follow instructions. It weeds out the people who just want an invite and don't care about communication outside of "LFG" in guild chat.
We had no problem with recruits who suck, or recruits who were awesome new guild members. The soul-grinding work were the people who sat on the fence. These people were often online very little, rarely spoke to people, or never really made any impressions on people. Some people suggesting asking the recruit to "shape up" to get them over the hump, but I am a firm believer that if they are a recruit and on their best behavior now, when they are a full member they will let their guard down and end up being a jerk. Asking them to change would just give me a false sense of security. Now we have a basic voting system. Recruits remain recruits until they get five positive votes. If they have more negative votes than positive after two weeks, they get the boot. If they have no negatives, and not quite five positive votes, they stay a recruit until they get the five. Those people who don't engage the guild or actively interact with the guild will remain a recruit for longer. How hard is it to get five people to agree you are a good fit for the guild? If it's that hard, maybe it's not the right guild for you.
We also have a members-only section of our forums where these recruits applications are posted. Full members are allowed to post their votes and their opinions about the recruits. There just isn't any way for the guild officers to really keep track of ten new people, so we use the regular members for feedback. In theory, these regular members are playing with these people and seeing them when the officers might not. Is the new recruit a jerk? Is he whining about loot? Do they have a great sense of humor? All great things to know when deciding if someone is going to be part of your guild.
Finally, you have to have good recruiters and members. Groups of people tend to attract similar people and your members are those people. You can not just invite everyone who is not in a guild and tell people to "just get along". Bringing in like minded individuals will strengthen your guild and your recruiters are the gate-keepers who make sure it happens. If the recruiters have a good feel for the guild and what the current guild is like, you'll get better quality people coming into the guild. If your recruiter was made a member one week ago and just wants to help make the guild "awesome", you're going to get crappy recruits.
So, in the end, what worked for us were a few things.
1) Higher bar of entry.
2) Recruit driven goals for membership.
3) Peer review.
4) Good recruiters and good guild members.
It's THAT easy!