It's not that the old man is the only one who belongs to the ghost house. But these people are deliberately hiding their identities. As for the old man in black robe, who can control four or five underworld yakshas, his identity is definitely not ordinary. This is probably why he did not hide his identity and destroy the wooden signs....