Since Sydney spent four billion euros ($5 billion) to put on the 2000 Games the amounts involved have skyrocketed. Beijing pumped around 40 billion into hosting the 2008 Games.
The London Olympics are expected to come in with a 12 billion euro price tag. Sochi 2014 is expected to be the most expensive winter games ever at around 24 billion euros.
While the blame is laid on the demands made by International Olympic Committee (IOC), host cities and nations often use the games as a pretext to launch massive infrastructure and construction projects.
Through a number of measures, such as limiting the number of athletes and sports, the IOC has succeeded in keeping the operating budgets of the games at around two billion euros.
This covers the two weeks of the Games themselves, the feeding and transporting of the athletes and operating the stadiums. Since 1984, costs have always been covered profitably from ticket sales and television and marketing rights.
The other part of the budget is for infrastructure. This can vary greatly, depending on whether host cities need to build lots of stadiums and transport infrastructure to handle the influx of spectators.
The Games are also often an opportunity for hosts to carry out long-needed redevelopment, which can increase the price tag even further.
"The Olympic Games are often a means to concentrate into two or three years the investments that would normally be spread out over 20 years," then IOC head Jacques Rogge told AFP in 2008.
China went forward on airport and road construction projects that the IOC hadn't considered as absolutely necessary to host the games, he said.
For the London Olympics, the infrastructure budget swelled because the British government used the preparations to redevelop huge swathes of East London, including thousands of new homes and transport links.
"This is a choice that we cannot criticise," Rogge said.
It did however triple the initial estimate of the price tag of the Games.
"Yes, the IOC has tried to keep down the costs," IOC executive director Gilbert Felli said recently about Rio di Janeiro redevelopment plans for the 2014 World Cup and 2016 Summer Games.
"But if a city wants to profit from the Games to build a new metro, rebuild half of its neighbourhoods, there's nothing you can do.
"It will be extremely expensive, but this isn't the IOC's business," he added.
Big, rapidly growing countries such as China and Brazil can shoulder the costs.
But for Britain, which has fallen back into recession as it tries to bring its public finances back into order, the party may have a hangover.
This was the case for Greece: the stimulus from the 2004 Games was small, but saddled the Greek government with debt.
"The effect on GDP (economic output) is minimal, even for Greece," said Jean-Louis Chappelet, a professor of public management at the University of Lausanne.
On the other hand, the roughly 10 billion euros it cost to put on the Games swelled Greece's debt by two to three percent, according to statements by IOC officials.
Worse, Athens is getting little use out of the facilities, which have for the most part been abandoned.
China is getting little use from its Bird's Nest Stadium although Beijing does not consider it a white elephant, but as the symbol of a successful Games.
But with its economy still growing at over seven percent China can afford letting the stadium stand as a tourist landmark visited by thousands everyday.