Serbia, competing for Olympic medals for the first time, will face China who are playing for their third gold after overcoming the Netherlands in four sets.
The Serbians were riding high after sending Russia home two days earlier and defeated the U.S. 20-25 25-17 25-21 16-25 15-13 in a five-set clash that could have gone either way.
The U.S. team won the first set but Serbia, driven on by their formidable spiker Brankica Mihajlovic, took control after America's top-scoring middle blocker Foluke Akinradewo went off with a left knee injury sustained in the opening set.
The Americans adjusted, battled back to win the fourth and were leading the tie-break before going down.
"We put ourselves in a position to win the match but Serbia made some great plays down the stretch," U.S. coach Karch Kiraly told reporters. "This loss is deeply disappointing. It cuts deep."
The U.S. were favorites to take their first gold medal in women's volleyball. They won the World Championship in 2014 and took silver in the last two Olympics, losing both finals to Brazil, who were eliminated by China on Tuesday.
It was a sad outcome for the U.S., who have risen to become the world's best and were within reach of the gold.
"We are going after the bronze, that's our goal now," said U.S. captain Christa Dietzen, with tears in eyes as she recalled the distraction caused to her team by Akinradewo's injury.
Kiraly said Akinradewo was in pain and could not continue. He did not detail the injury, but reporters saw her pressing an ice pack to her left knee.
Serbia now face China, who they beat in the preliminary round, in the final after the Chinese defeated a wily Dutch team in a fiercely fought match that ended 27-25 23-25 29-27 25-23.