A seriously depleted United looked to have given themselves too much to do after losing 2-0 in the first leg of their last-16 tie last month, yet a Romelu Lukaku brace in the first half at the Parc des Princes gave them hope.
Even so, Juan Bernat had netted in between for PSG, and it looked as though the French side would stumble on until the drama at the death.
Slovenian referee Damir Skomina awarded a spot-kick after reviewing the images when he had been alerted of a possible handball by PSG defender Presnel Kimpembe in the box.
Rashford duly beat Gianluigi Buffon from 12 yards, as United won a European tie after losing the first leg at home for the first time in their history.
“We always believed, that’s the thing,” an elated Solskjaer said. “The plan was to get the first goal, be in the game with five or 10 minutes to go, and we were.”
Their improbable victory – following Ajax’s similarly stunning turnaround against holders Real Madrid 24 hours earlier – takes them through to the last eight for the first time since 2014.
“Of course we fancy ourselves. We can go all the way, but we just have to wait for the draw and then take the game as it comes,” said the Norwegian, who has now overseen nine consecutive away wins, a club record.
“We have had loads of injuries and suspensions but that was maybe a good thing today because we had a fresh team and loads of energy.”
While United celebrate, for PSG and their Qatari owners this defeat will feel like groundhog day.
For the third year running, they fail to make the quarter-finals, after their defeat at the hands of Real Madrid a year ago and their humiliating collapse against Barcelona in 2017.
Electric at Old Trafford in the first leg, Kylian Mbappe was poor here and Neymar was again missing, watching from the stands as he recovers from injury.
“If it were a league game you would wipe your mouth and go on, but today it’s horrible and cruel,” said coach Thomas Tuchel.
“It happens once in 100 games. We did not deserve to go out over 180 minutes.”
However, the German refused to say the officials had got the late penalty call wrong.
“I think it’s a 50-50 decision. What makes it super hard is that I had the clear feeling that the shot was not on target.”
United were missing Paul Pogba through suspension and nine more players were sidelined due to fitness problems.
Despite that, they achieved one of their greatest European results, up there with their 3-2 win at Juventus in the 1999 semi-finals, which came after they fell two goals behind early on.
Solskjaer was on the bench that night, and the Norwegian watched from the sideline here – curiously with a bib on over his jacket apparently due to a colour clash with PSG’s kit – as his side went in front inside two minutes.
A dreadful back-pass attempt by PSG defender Thilo Kehrer fell to Lukaku, who rounded Buffon and scored.
The hosts pulled themselves together to equalise in the 12th minute when Mbappe squared for Bernat to turn in his third Champions League goal this season.
However, the visitors were gifted another goal half an hour in.
At 25 yards out, Rashford tried a shot that was powerful but straight at Buffon. Yet the veteran Italian spilled the ball, and Lukaku followed in to score his sixth goal in three games.
PSG – so slick in France – were wobbling again at the business end of the Champions League.
Angel Di Maria had a goal ruled out for offside in the 56th minute and Mbappe then managed to fall when clean through late on, with Bernat sending the loose ball against the post.
It still looked as though they would hang on, but then VAR intervened, and Rashford sent United into ecstasy.