Former world No.1 Williams was pulled all over the court by high-quality groundstrokes from Kenin, who received a walkover in round two when Bianca Andreescu withdrew because of a shoulder injury in the opening set.
Williams, who has been struggling with a left knee problem, let out huge roars as she attempted to work her way back into the match.
However, despite having the crowd on Court Philippe-Chatrier firmly behind her, the 37-year-old was unable to turn the tide against her fellow American and remains one Major title short of matching Margaret Court's all-time record.
Williams pulled out a couple of aces as she denied Kenin on three break points, but a sublime drop shot in game five saw her lose her footing and fall behind.
The No.10 seed missed a chance to hit straight back and was two breaks down when her Moscow-born opponent wrongfooted her with an exceptional forehand into the corner.
Kenin thought she had the first set wrapped up when Williams produced a long return, but her serve was called wide. However, she staved off a break point and another fine forehand preceded Williams netting a return to surrender the opener.
The world No.35 broke to love at the start of the second set and kept a fired-up Williams, who is a three-time champion at Roland Garros, at bay to consolidate.
Kenin was whistled for kicking the mark left by a cross-court backhand that just clipped the line and set up a break point for Williams, who was soon back on serve.
Boos rang out when Kenin questioned another line call but she was in position to serve out the match after a sensational forehand return in game 11.
Williams miscued wide on a chance to break back, while Kenin found the net on her first match point.
A long backhand brought about the end for Williams, with an emotional Kenin advancing to the last 16 of a grand slam for the first time, with Ashleigh Barty standing between her and a place in the quarter-finals.