Pretty much it right there. It depends on how recent it happened in relation to the searches being performed (enough time for survivors to be recovered or just bodies), and the type of disaster. They are supposed to use green (Inspected, structure sound), yellow (Restricted, up to medium damage) and red (unsafe, severely damaged and little if any exits other than the entrance) spray paint also. The different colours help easily identify how sound the structure is, but in recent times (Katrina, Rita, Ike) they are often rushed and just use other colours to make more rushed searches. These started off their life in earth quake damaged area's, and now are a common call sign for hurricane area's. Technically they follow the same law as those little inspection papers they stick to front doors of condemned buildings and can only be removed by one of the inspectors... but no one ever really does that and usually paint over them.
Usually they do quick searches for survivors in the off colours and then come back later and do more through searches (as in Katrina) and use proper colour coded tags then or proper colour coded paper search forms (same colour's used in forms) stapled to the front door (if present).
The "X" originally showed when it was searched, which task force searched it, which hazards are present, and how many victims are trapped (or dead) inside. Each task force consisted of two 31 person teams, cadaver dogs, their own gear/equipment unique to each teams needs. Now a simpler version is being used though as there are not enough structural engineers/cadaver dogs to be all over the place in such large area's.