In a dramatic late-night resignation speech from Washington, Rudd said the only honourable course of action was for him to step down, as reports circulated in Australia that Gillard was preparing to sack him.
"The simple truth is that I cannot continue to serve as foreign minister if I don't have Prime Minister Gillard's support," he told reporters at a press conference broadcast live in Australia.
Australia's Labor government has been torn by speculation about whether Rudd, who Gillard suddenly ousted as a prime minister in mid-2010 but who remains hugely popular with voters, would mount a bid to return to the top job.
The globe-trotting diplomat said he would touch down in Australia on Friday, and would consult with his family, community and parliamentary colleagues on what he would do next.
"I will then make a full statement to the Australian people on my future before parliament resumes next Monday," he said at the press conference at which he declined to take questions.
Rudd said the leadership speculation had become a "soap opera" which was a distraction from the real business of government.
In a swipe at the power of the "faceless men" in Australian politics, a reference to the factional leaders of the Labor Party who backed his ousting, he said: "Australia must be governed by the people, not by the factions."
He also insisted there should never be a "stealth attack" on a sitting prime minister elected by the people, a reference to his June 2010 deposing by his then-deputy Gillard who challenged him for the leadership, prompting his resignation as prime minister.
His brutal ousting took most Australians by complete surprise, and still stirs some sympathy among voters for the way he has been treated.
"We all know that what happened then was wrong and must never happen again," Rudd said.
Earlier Wednesday, The Australian newspaper reported that Gillard was confident she had the numbers among Labor MPs to hold onto the leadership and would call a ballot on Tuesday to force Rudd to put his hand up.
If he lost the ballot, Rudd would be immediately sacked as foreign minister by Gillard and moved to the backbench, it said.
The prime minister refused to comment on the report, saying she had already answered many questions about the leadership and telling reporters: "I really don't have much to add."
Labor has been wracked by internal divisions as Gillard struggles against unpopular policies and miserable polling that suggest the party would be unceremoniously dumped if an election held tomorrow.
As tensions mounted and a video of Rudd uttering expletives and losing his cool posted anonymously on YouTube went viral, former Labor leader Simon Crean called on the foreign minister to "put up or shut up".
Labor figures have called for an end to the constant speculation, with some saying there was no room in the cabinet for both Rudd and Gillard.
"Just having him there has been destabilising," one unnamed minister told The Australian's online edition. "It can't go on."
Rudd became prime minister in 2007, after a landslide electoral win which ended more than a decade of conservative rule. But he was ousted in June 2010 in a party-room vote after losing the support of Labor's factional leaders.
Gillard leads a fragile minority government after failing to secure Labor a majority at a 2010 election and Industry Minister Greg Combet said the leadership issue was a distraction which had to end.
"Enough is enough. It is time this matter is resolved," Combet said.