In "The Minister's Black Veil" Hawthorne is using the black veil as a symbol for the sins of man that are kept within each of us. Essentially, the evil that we are all capable of. In this story, Mr. Hooper has his own sin as well, and the black veil could be taken to mean literally his specific crimes against the soul. The townspeople are upset by this as they should be upset by sin (in Hawthorne's eyes).
