what is the official procedure if you answer "yes" to any of the US visa (DS 160) "are you a terrorist/slaver/bodysnatcher" questions?

What does actually happen if someone were hypothetically to state here that he's participated in genocide or stolen body parts?

If a visa applicant answers yes to any of these questions, the applicant is found to be inadmissible to the United States, the application is denied, and no visa is issued. The applicant's passport would normally be returned, though I suppose it's possible that the applicant might be reported to law enforcement authorities in the host country, in which case the applicant's passport might not be returned.

I'm asking if there is any kind of official procedure, government guideline or even law that governs that.

The law, as noted in another answer, is 8 USC 1182, which governs inadmissibility.

Aside from the obvious cases, there's a few that might actually come up, such as the question about prostitution in the past 10 years (for people from countries where prostitution is legal).

8 USC 1182(a)(2)(D)(i) provides that a person who has (among other things) "engaged in prostitution within 10 years of the date of application for a visa" is inadmissible, without regard to the legal status of the act of prostitution in the jurisdiction in which it occurred, so it doesn't matter where the person comes from or where the person engaged in prostitution. As with the other inadmissibility questions, the wording is essentially taken directly from the statute, so a "yes" answer would lead automatically to denial.

As to admissions of espionage, sabotage, or terrorism, it's probably likely that the CIA would learn of them, since it's fairly common knowledge that the CIA, like most intelligence services, places its agents in diplomatic and consular posts. The CIA or the State Department might also alert the intelligence services or law enforcement authorities of the host nation; I don't know whether policies governing that would be publicly available.