Trailing 2-0 in Washington, Latvia's fightback started with Jelena Ostapenko's win over Australian Open champion Sofia Kenin.
Williams's unbeaten record in singles was then ended by Anastasija Sevastova, who recorded a shock 7-6 (7-5) 3-6 7-6 (7-4) victory over the 23-time grand slam champion.
But United States moved into April's Fed Cup Finals thanks to Kenin and Bethanie Mattek-Sands, the duo winning the doubles over Ostapenko and Sevastova 6-4 6-0.
Belarus needed a thrilling doubles rubber to overcome Netherlands 3-2.
Aryna Sabalenka and Aliaksandra Sasnovich edged Kiki Bertens and Demi Schuurs 4-6 6-3 7-6 (10-8).
Russia survived a test against Romania to prevail 3-2, Switzerland claimed a 3-1 success over Canada and Slovakia beat Great Britain 3-1.
Germany swept Brazil aside 4-0, Spain was too strong for Japan 3-1 and Elise Mertens led Belgium past Kazakhstan 3-1.