Osaka, the 2018 US Open champion, needed 2hr 33min to beat the unseeded world number 137 6-3, 6-7 (4/7), 6-2 inside an empty Arthur Ashe Stadium.
"She was very good. I'm kind of scared of how she is going to be in the future," Osaka said of her vanquished opponent.
Frustration got the better of the Japanese fourth seed in the second set. After recovering from 5-3 to force a tie-break she threw her racquet when Kostyuk tied the match 1-1.
"I just felt like I had so many points that I didn't capitalise on. I think when I went up, I became a bit passive and then she came in because she has no fear," Osaka said.
But Osaka's experience came through in the deciding set as she held her serve and broke Kostyuk twice to set up a match against either 14th seed Anett Kontavelt of Estonia or Poland's Magda Linette of Poland for a place in the quarter-finals.
"It was probably experience. I know she's very young and I think I have (played) more matches that are like this," she said.
Osaka, of Haitian and Japanese heritage, walked onto the court wearing a facemask emblazoned with the name of Ahmaud Arbery, an unarmed black man who was shot dead in broad daylight while jogging in the town of Brunswick, in southern Georgia, in February.
Osaka is wearing different masks honoring victims of racial injustice and police brutality throughout the tournament.
"I would like everyone to know that it was completely avoidable. This did not have to happen. None of these deaths have to happen and for me, I feel like I just want everyone to know the names," she said.
Ahead of her second-round win on Wednesday she wore a covering with the name Elijah McClain, a 23-year-old African-American man who died in police custody last year after being placed in a chokehold and injected with a sedative.
On Monday, Osaka, the 2019 Australian Open champion, honored Breonna Taylor, the African-American nurse shot dead by police who raided her apartment in Kentucky in March.