Mr Cameron made the announcement in a statement outside Downing Street after the final result was announced.
He said he would attempt to “steady the ship” over the coming weeks and months but that “fresh leadership” was needed.
The PM had urged the country to vote Remain, warning of economic and security consequences of an exit, but the UK voted to Leave by 52% to 48%.
England and Wales voted strongly for Brexit, while London, Scotland and Northern Ireland backed staying in.
UKIP leader Nigel Farage hailed it as the UK’s “independence day” but the Remain camp called it a “catastrophe”.
The pound fell to its lowest level against the dollar since 1985 as the markets reacted to the results.
Flanked by wife Samantha, Mr Cameron said he had informed the Queen of his decision to remain in place for the short term and to then hand over to a new prime minister by the time of the Conservative conference in October.
It would be for the new prime minister to carry out negotiations with the EU and invoke Article 50 of the Lisbon Treaty, which would give the UK two years to negotiate its withdrawal, he said.
“The British people have voted to leave the European Union and their will must be respected,” said Mr Cameron. “The will of the British people is an instruction that must be delivered.”
The referendum turnout was 71.8% – with more than 30 million people voting – the highest turnout at a UK-wide vote since 1992.
Labour’s Shadow chancellor John McDonnell said the Bank of England may have to intervene to shore up the pound, which lost 3% within moments of the first result showing a strong result for Leave in Sunderland and fell as much as 6.5% against the euro.
UKIP leader Nigel Farage – who has campaigned for the past 20 years for Britain to leave the EU – told cheering supporters “this will be a victory for ordinary people, for decent people”.
Mr Farage – who predicted a Remain win at the start of the night after polls suggested that would happen – said it would “go down in history as our independence day”.
He called on Prime Minister David Cameron, who called the referendum but campaigned passionately for a Remain vote, to quit “immediately”.
Area-by-area in maps: See how people voted
Labour sources also said David Cameron “should seriously consider his position”.
But pro-Leave Conservatives including Boris Johnson and Michael Gove have signed a letter to Mr Cameron urging him to stay on whatever the result.
Labour leader Jeremy Corbyn, who called for the UK to remain in the EU but was accused of a lukewarm campaign, said poorer communities were “fed up” with cuts and felt “marginalised by successive governments”.
“Clearly there are some very difficult days ahead,” he said, adding that “there will be job consequences as a result of this decision”.
He said the point he had made during the campaign was that “there were good things” about the EU but also “other things that had not been addressed properly”.
Former Labour Europe Minister Keith Vaz told the BBC the British people had voted with their “emotions” and rejected the advice of experts who had warned about the economic impact of leaving the EU.
He said the EU should call an emergency summit to deal with the aftermath of the vote, which he described as “catastrophic for our country, for the rest of Europe and for the rest of the world”.
Germany’s foreign minister Frank Walter Steinmeier described the referendum result as as “a sad day for Europe and Great Britain”.
But Leave supporting Tory MP Liam Fox said voters had shown great “courage” by deciding to “change the course of history” for the UK and, he hoped, the rest of Europe.
And he called for a “period of calm, a period of reflection, to let it all sink in and to work through what the actual technicalities are,” insisting that Mr Cameron must stay on as PM.
Scotland’s First Minister Nicola Sturgeon has said that the EU vote “makes clear that the people of Scotland see their future as part of the European Union” after all 32 local authority areas returned majorities for Remain.
Analysis by Prof John Curtice
London has voted to stay in the EU by around 60% to 40%.
However, no other region of England has voted in favour of remaining.
The referendum has underlined the social and cultural gap between London and provincial England.
Remain’s defeat seems to have been primarily the product of the decisions made by voters living north of the M4.
Throughout the Midlands and the North of England the level of support for Remain was well below what was required for it to win at least 50% of the vote across the UK as a whole.
Britain is set to be the first country to leave the EU since its formation – but the Leave vote does not immediately mean Britain ceases to be a member of the 28-nation bloc.
That process could take a minimum of two years, with Leave campaigners suggesting during the referendum campaign that it should not be completed until 2020 – the date of the next scheduled general election.
The prime minister will have to decide when to trigger Article 50 of the Lisbon Treaty, which would give the UK two years to negotiate its withdrawal.
Once Article 50 has been triggered a country can not rejoin without the consent of all member states.
Mr Cameron has previously said he would trigger Article 50 as soon as possible after a Leave vote but Boris Johnson and Michael Gove who led the campaign to get Britain out of the EU have said he should not rush into it.
But they also said they want to make immediate changes before the UK actually leaves the EU, such as curbing the power of EU judges and limiting the free movement of workers, potentially in breach the UK’s treaty obligations.
The government will also have to negotiate its future trading relationship with the EU and fix trade deals with non-EU countries.
In Whitehall and Westminster, there will now begin the massive task of unstitching the UK from more than 40 years of EU law, deciding which directives and regulations to keep, amend or ditch.
The Leave campaign argued during a bitter four-month referendum campaign that the only way Britain could “take back control” of its own affairs would be to leave the EU.
Leave dismissed warnings from economists and international bodies about the economic impact of Brexit as “scaremongering” by a self-serving elite.