The American great was playing her first competitive tournament since the 2017 Australian Open after giving birth to her daughter in September.
After beating Zarina Diyas and Kiki Bertens in her first two matches, the third-round encounter against her sibling proved a task too far as Venus ran out a 6-3 6-4 winner.
Speaking after her win, Serena recognised that she has work to do to return to the top of the game but is relishing the challenge.
"I haven't played in over a year," she said. "[Losing is] definitely not less disappointing. I wish it were, but it's not. But then again, I wish it wasn't. Then I wouldn't be who I am.
"So, I just have a long way to go, and I'm looking forward to the journey."
Venus was impressed by her younger sister's standard of play and believes the fact she has played more competitive matches was a decisive factor.
“[Serena's] level is super high and it was very difficult to close out the match, just getting one ball back," Venus said. "I think just I have played more matches in the last year. Even though I haven't even played that much this year, the matches in the last year count."