Gingrich compared supporters of the mosque to Nazis. Appearing on Fox & Friends, Gingrich said, "Nazis don't have the right to put up a sign next to the Holocaust museum in Washington."

Prior to that, Gingrich argued that the mosque shouldn't be built near Ground Zero until churches and synagogues are allowed in Saudi Arabia.

"This is demagoguery of the first order," Scarborough said Monday. "And people in the Republican Party need to separate themselves from these voices."

"And I talk to you, my Republican brethren," he said into the camera. "I don't know how much longer you'll be my brethren. I'll be honest. I'm looking for a conservative party that actually believes in small government and not engaging in Wilsonian wars but that's another discussion."

"I'm just talking, you know, as a friend," Scarborough continued. "I promise you this. You're going to be embarrassed. You're going to look back two, three, four years from now and this is going to be dark blot on your record if you don't speak out against New Gingrich and the voices of hate."

"This is an embarrassment and you need to speak out against it," he said.

Opponents of the mosque protested in lower Manhattan Sunday. Daisy Khan, the wife of the controversial imam backing the Islamic center, said Sunday that opposition was "like a metastasized anti-Semitism."

MSNBC's Willie Geist told Scarborough Monday that the opposition is proof that anti-Muslim sentiment is worse now than after Sept. 11, 2001.

"It shows us that we are probably farther backward that we were maybe even nine years ago in our interfaith relations," said Geist.