Crowds paid tribute, dancing and singing in front of Mr Mandela’s former home in Soweto throughout the night.
Flags flew at half mast after President Jacob Zuma announced his death in a late night national TV address.
Mr Mandela spent 27 years in jail before becoming South Africa’s first black president in 1994.
His administration replaced the racist white-minority regime that had enforced segregation of black and white people in a policy known as apartheid.
Mr Mandela went on to become one of the world’s most respected statesmen.
A service of national mourning is expected to be held at a 95,000-seater stadium on the outskirts of Johannesburg on Monday. His body will then lie in state for three days in the capital, Pretoria, before being taken for a state funeral in the village of Qunu in the Eastern Cape, where he grew up.
“God was so good to us in South Africa by giving us Nelson Mandela to be our president at a crucial moment in our history,” said long-time ally Archbishop Emeritus Desmond Tutu.
At a service in Cape Town on Friday, he said Mr Mandela had “taught a divided nation to come together”.
Mr Mandela had been suffering from a lung illness for a long time.
He had been receiving treatment at home since September, when he was discharged from hospital.
As soon as the news broke, small crowds began to gather in Soweto’s Vilakazi Street, where Mr Mandela lived in the 1940s and 1950s.
They chanted apartheid-era songs, including one with the lyrics: “We have not seen Mandela in the place where he is, in the place where he is kept.”
By daybreak, dozens more had gathered.
“We are celebrating his life and all that he did for us,” said one of the mourners, Terry Mokoena.
Crowds also gathered outside Mr Mandela’s current home, in Johannesburg’s northern suburb of Houghton, where he died.
Across the world, leaders, celebrities and members of the public have been paying tribute.
Queen Elizabeth II said she was “deeply saddened” to learn of Mr Mandela’s death.
“He worked tirelessly for the good of his country, and his legacy is the peaceful South Africa we see today,” a statement issued by Buckingham Palace said.
“Her Majesty remembers with great warmth her meetings with Mr. Mandela and sends her sincere condolences to his family and the people of South Africa at this very sad time.”
“He achieved more than could be expected of any man. Today, he has gone home,” said US President Barack Obama.
UN Secretary General Ban Ki-moon called him “a giant for justice and a down-to-earth human inspiration”.
Flags are flying at half-mast on government buildings in Washington DC, Paris and across South Africa. World football body Fifa also ordered its flags to be lowered.
The parliament in Pretoria is expected to hold a special joint session to reflect on Mr Mandela’s life and legacy.
Books of condolence will be opened at public buildings in South Africa and at the country’s embassies throughout the world.
Next Monday is expected to be the start of South Africa’s official mourning, with a service in Soweto’s FNB stadium.
Mr Mandela’s body will lie in state for three days at the Union buildings in Pretoria before a funeral is held on Saturday in Qunu, the village in Eastern Cape where he was born.
The BBC’s Mike Wooldridge in Johannesburg says South Africa will never have seen a state funeral like it, with leaders, dignitaries and other admirers of the former president expected from all over the world.
It will be a huge logistical challenge, especially given the remoteness of Qunu, our correspondent adds.
In his TV address, Mr Zuma said Mr Mandela had died shortly before 21:00 local time (19:00 GMT).
“Our nation has lost its greatest son,” Mr Zuma said.
“Although we knew that this day would come, nothing can diminish our sense of a profound and enduring loss.”
Mr Mandela won admiration around the world when he preached reconciliation after being freed from almost three decades of imprisonment.
He was awarded the Nobel Peace Prize in 1993 along with FW de Klerk, South Africa’s last white president.
Mr de Klerk, who ordered Mr Mandela’s release from jail, called him a “unifier” and said he had “a remarkable lack of bitterness”.
He told the BBC that Mr Mandela’s greatest legacy was that “we are basically at peace with each other notwithstanding our great diversity, that we will be taking hands once again now around his death and around our common sadness and mourning”.
Mr Mandela had rarely been seen in public since officially retiring in 2004. He made his last public appearance in 2010, at the football World Cup in South Africa.
Born in 1918, Nelson Mandela joined the African National Congress (ANC) in 1943, as a law student.
He and other ANC leaders campaigned against apartheid. Initially he campaigned peacefully but in the 1960s the ANC began to advocate violence, and Mr Mandela was made the commander of its armed wing.
He was arrested for sabotage and sentenced to life imprisonment in 1964, serving most of his sentence on Robben Island.
It was forbidden to quote him or publish his photo, but he and other ANC leaders were able to smuggle out messages of guidance to the anti-apartheid movement.
He was released in 1990 as South Africa began to move away from strict racial segregation – a process completed by the first multi-racial elections in 1994.
Mr Mandela served a single term as president before stepping down in 1999.
After leaving office, he became South Africa’s highest-profile ambassador, campaigning against HIV/Aids and helping to secure his country’s right to host the 2010 football World Cup.
He was also involved in peace negotiations in the Democratic Republic of Congo, Burundi and other countries in Africa and elsewhere.