The nation's first female prime minister retained office after smashing Rudd 71-31 in a secret ballot of the 103-member Labor caucus, although only 102 votes were ultimately cast with one member absent.
"The ballot has taken place and Julia Gillard has won the ballot 71 votes to 31," said returning officer Chris Hayes of an acrimonious row.
"I have just formally declared Julia re-elected as leader of the parliamentary Labor Party. The mood I think it is fair to say was reasonably tense," he added.
It was among the biggest-ever wins in a Labor leadership ballot.
Gillard called the vote in a bid to end a bitter standoff with her predecessor, whom she dumped as leader in 2010 in a shock party coup, and bring to a head a period of intense turmoil within Labor.
Rudd came to power in a 2007 election landslide that ended more than a decade of conservative rule, but a series of policy mis-steps saw him lose the confidence of party chiefs and he was axed for the more pragmatic Gillard.
Observers say he never forgave Gillard and he dramatically quit as foreign minister last week before announcing a challenge for the top job, believing only he could save Labor from electoral annihilation in 2013.
Moving forward, Gillard will almost certainly reshuffle her cabinet after several of her ministers came out in support of Rudd, including Resources Minister Martin Ferguson and Immigration Minister Chris Bowen.
She also has to announce a replacement for Rudd as foreign minister, with Defence Minister Stephen Smith, who preceded Rudd in the portfolio, seen as the favourite.
Rudd will now retire to the backbench and has pledged "unequivocal support" for Gillard, with both camps urging unity to try and repair the damage done to the party.
Speaking shortly before the vote, he reiterated that he would not mount another challenge if he lost.
"I won't be initiating any (further) challenge against Julia," said Rudd, who has been subjected to scathing criticism from fellow ministers who have described him as "dysfunctional".
Yet Rudd remains popular outside the caucus, with his resounding defeat coming as a Newspoll for The Australian, taken over the weekend, showed 53 percent of voters favour him as prime minister to Gillard's 28 percent.
Rudd is also far more popular than opposition leader Tony Abbott, with the poll of 620 people showing 53 percent think he would be a better prime minister to Abbott's 34 percent.
Support for Gillard, meanwhile, has declined, with her satisfaction rating falling from 32 percent to 26 percent in the past fortnight.
But in a glimmer of hope for Labor, backing for the party is at its highest in 12 months, rising three points to 35 percent from two weeks ago compared to the conservative coalition, which is steady at 45 percent.
While Labor has vowed unity after vote, the poor polling of Gillard saw staunch Rudd supporter, backbencher Doug Cameron, warn of a second leadership challenge within months unless her standing with voters improves.
"If ... we end up in the same position as we are now, in terms of the polls in several months time, then my view is the same people who installed Julia Gillard will be looking for a candidate to replace Julia Gillard," he said.
Others have downplayed this scenario and urged all Labor members to get behind Gillard.
"We have to get on with the job of being united and running the country," Smith told Sky News.