Rep. Donna Edwards (D-MD) explained to retired conservative neurosurgeon Ben Carson on Sunday that he was making a mistake by refusing to talk about race because there are "deep systemic problems that really impact African-Americans."

Fox News Sunday invited Carson -- who has been talked about as a possible 2016 Republican presidential candidate -- to respond to President Barack Obama's surprise remarks on Friday about how his experience as a black man related to the Trayvon Martin case.

"We have a tendency to overemphasize the superficial," Carson told Fox News host Chris Wallace. "You know, I was asked once by an NPR reporter why I don't talk about race that often. And I said it's because I'm a neurosurgeon. And she looked at me quite quizzically. And I said, 'You see, when I take someone to the operating room and I peel down the scalp and take off the bone flap and open the dura, I'm operating on the thing that makes the person who they are.' It's not the covering that makes them who they are."

"We need to be looking for things that we can take out of this situation that will be helpful, not things that inflame the situation," he added. "Let's tone down the rhetoric and recognize that we, the people, are not each others' enemies."

But Edwards pointed out that not all African-Americans had access to the same benefits that had helped Carson to be so successful.

"But Dr. Carson, really to be clear, there really are deep systemic problems that really impact African-Americans in this system that have to be addressed and I think the president spoke to some of those," Edwards said. "And so, we can't ignore that."

"And I'm going to acknowledge like anyone else that everything is not about race but there are some things that are, and we have to acknowledge those and we have to make the systemic changes that are important to advance all of our communities so that people really can enjoy those things and the benefits, frankly, that Dr. Carson has enjoyed."

Earlier this year, Carson did speak out about race when he slammed "white liberals" as being the true racists.

"They’re the most racist people there are," he told conservative radio host Mark Levin. "Because they put you in a little category, a box: ’You have to think this way, how could you dare come off the plantation?’"