On Fox News, the former Utah Congressman was worried that some ladies "will want to take advantage and throw an allegation" around to ruin you.