The disappointment was compounded by Mourinho's controversial decision to drop goalkeeper and captain Iker Casillas from the starting line-up.
It was the first time in 10 years that the Spanish World Cup winner had been axed with 25-year-old Antonio Adan called up to take the place of the benched veteran.
Isco and Roque Santa Cruz, with a double, grabbed the goals for Malaga.
Madrid did level at 1-1 with an own-goal by Samuel Sanchez and Karim Benzema made it 3-2 eight minutes from time, but the home side clung on to celebrate their first win over Real in 29 years.
Malaga's win left the Costa del Sol club in fourth, two points behind Madrid on 31 points while Atletico Madrid, who beat Celta Vigo 1-0 on Friday, are second, nine points behind Barcelona.
Barca had earlier won 3-1 at Valladolid, a timely boost for cancer-stricken coach Tito Vilanova who left hospital Saturday 48 hours after undergoing surgery to remove a tumour in his saliva gland.
Mourinho was admamant that he would not quit the Real hotseat.
“I'm not worried about my job at all and I'm not going to resign,” said the Portuguese, who justified his decision to axe Casillas.
“At the moment, for me and my coaching staff, Adan is better than Iker. We have been struggling in a number of areas, one of those has been defence. The coach analyses who is in the best situation to play and chooses his team.”
Isco opened the scoring for Malaga after 48 minutes with a well-placed shot from just outside the area, that Adan could do nothing about, after Joaquin Sanchez had found space on the left.
Mourinho brought on Kaka on 63 minutes and his side equalised immediately when the Brazilian's cross caused chaos in the home defence and Sanchez turned the ball into his own goal.
However Malaga's attacking play was justly rewarded with two further goals, both from the same source.
On 72 minutes, Joaquin's clever back-heel found ex-Manchester City forward Santa Cruz, who had replaced former Real star Javier Saviola, in front of goal and three minutes later the Paraguayan finished well from a clever Joaquin pass.
Benzema gave Madrid hope but Mourinho's side never looked like getting anything from the game in the closing stages.
Xavi Hernandez, Lionel Messi and Cristian Tello steered La Liga leaders Barcelona to victory at Valladolid.
Jordi Roura took charge of Barca in the absence of the 44-year-old Vilanova who will be out of the game for a number of months to undergo radiotherapy and chemotherapy.
The win allowed Barcelona to finish 2012 with with 16 wins in 17 games this season, the 2-2 home draw with Real Madrid being the only game in which Barca have dropped points.
“The result could not be more satisfying. We have ended the year very well in a sporting sense and if we add to this the good news about Tito, we can be very happy”, said Roura.
“The only way we had to help him was winning the game and, once again, the team's attitude has been remarkable.”
Alexis Sanchez should have opened the scoring as early as the fifth minute, but after running onto a Dani Alves pass, he shot wide with the goal at his mercy.
Valladolid goalkeeper Dani Hernandez was called into action on 11 minutes to stop a shot from Messi and Xavi went close with a curling free-kick after 26 minutes.
Messi hit a free-kick against the post on 35 minutes after centre-half Henrique Sereno had blocked Thiago Alcantara on the edge of the area.
Barca took the lead with three minutes of the half remaining when a quick attack saw Messi find full-back Jordi Alba in an advanced position on the left and his cross left Xavi with the easiest of finishes.
Messi doubled the lead with a stunning goal on the hour mark.
The Argentine put the ball through a defender's legs during a driving run before hitting a crisp left-footed shot into the corner to finish 2012 on a staggering 91 goals.
The capacity 26,500 crowd sportingly applauded Andres Iniesta onto the pitch as a late substitute before Javi Guerra gave the home side a late consolation from close range.
But substitute Tello made it three for Barca with his only touch of the game in injury time.