The man Trump once called "my African-American" is no longer in Trump's corner. Here's why.