Los Angeles Lakers star James hit back and vowed never to "just stick to sports", insisting he had a role to play as a voice against racism and other pressing societal issues.
James is a friend of former United States president Barack Obama and his own foundation supports a school that is aimed at helping disadvantaged children.
But Ibrahimovic, who has also been criticized for agreeing to appear in the Sanremo music festival amid the Serie A season, refused to back down.
"Racism and politics are two different things. Athletes unite the world, politics divides it," said the 39-year-old, who was injured in Sunday's 2-1 win over Roma and could be sidelined for up to three weeks, which would rule him out of facing former club Manchester United in the Europa League later this month.
"Everyone is welcome in our environment, it doesn't matter where you come from and we are doing everything to bring people together.
"My message? Athletes should be athletes, politicians should be politicians."
Responding to questions over his appearance at the music event, which spans four days, while Milan fight for the title, the Swede added: "I'm a professional and anyone who knows me, knows that. When I play football I'm only focused on that.
"I want to help Milan, and to give a lot to Italy for everything that it has given me over the years, not only in football.
"I had the chance to be a guest at the festival, one of the most important in Italy, and decided to participate."