Ms Andersson was the Nordic nation's first female PM when she took office last year - she quit on the first day, before returning soon after.
She has been edged out by a four-party right-wing block made up of the Sweden Democrats, Moderate Party, Christian Democrats and Liberals.
It is a momentous turning point for Swedish politics - the Sweden Democrats was once treated as a pariah by political parties, but has now won around 20% of the vote.
It vowed to "make Sweden safe again" by bringing in longer prison sentences and restricting immigration.
However the party's leader Jimmie Akersson will not become prime minister, the BBC's Maddy Savage reports from Stockholm, because he does not have the full support of all four parties, according to BBC.
Instead, Ulf Kristersson, leader of the Moderates has said he will start to form a government.
"I am ready to do all I can to form a new, stable and vigorous government for the whole of Sweden and all its citizens," Mr Kristersson said on Wednesday.
The Sweden Democrats party was born out of a neo-Nazi movement at the end of the 1980s, but has slowly grown stronger while attempting to polish its image.
In 2019 Mr Kristersson started talks with the party to form an alliance - a move which has now changed Sweden's political landscape.
Ms Andersson told reporters on Wednesday that she understood those concerned at the party's growing popularity. "I see your concern and I share it," she said.
Magdalena Andersson's Social Democrats had governed Sweden since 2014 and dominated the country's political landscape since the 1930s.
Sunday's vote was one of the closest elections ever in Sweden, with thousands of overseas and postal votes needed to be counted to get a clearer picture of who the winner was, BBC reported.
