Saturday's match in Wakayama was tied at 1-1 when abandoned with five minutes to go, moments after Felix Uduokhai had cancelled out Douglas Martinez's first-half opener.
The Germany national team's official Twitter account later confirmed the reason for the game, which was split into three 30-minute sessions, being cut short.
"The game has ended five minutes early with the score at 1-1," the tweet read. "The Germany players left the pitch after Jordan Torunarigha was racially abused."
Torunarigha plays for Hertha Berlin at club level and has played for Germany from Under-16s to Under-23s level.
The 23-year-old was also the alleged victim of racist abuse in February 2020 in a DFB-Pokal match between Hertha and Schalke.
Following the latest incident on Saturday, Hertha offered their support to the centre-back, tweeting of the decision to leave the pitch in unison: "That is the only right decision!"
Germany face Brazil on July 22 in their opening Group D fixture at the Olympics, before taking on Saudi Arabia and Ivory Coast.