Two ex-Pompey players combined to punish their former club for a poor second half showing at the New York Stadium as Joe Mattock headed in Michael Smith's cross at the death.
The visitors edged a poor first half where both teams were guilty of giving the ball away on far too many occasions. Brett Pitman had the best of Pompey's chances, placing a weak header into Marek Rodak's arms following a counter-attack when a stronger effort would have resulted in a goal.
David Ball came closest for Rotherham as his effort sailed narrowly over the bar but both teams couldn't capitalise on the others' mistakes and therefore both goalkeepers were left relatively untroubled.
The second half was a totally different story as the hosts totally dominated and forced a succession of corners which were all scrambled away in various different ways, although the ball almost always came back at the Pompey defence as there was no outlet to keep the ball away.
There was one good chance for Pompey to sucker punch their promotion rivals when a counter-attack ended with Jamal Lowe sending the ball over the bar from 18 yards when he could have played in Kal Naismith who was in a far better position to shoot.
Instead, it was Rotherham who bagged a late winner thanks to Mattock's header at the far post and whilst the goal was obviously met by vastly contrasting emotions from both teams, it had been coming ever since the whistle to start the second half blew.
Kenny Jackett has to shoulder much of the blame for yesterday's defeat as his second half tactics simply didn't work for the second week running.
Rotherham quickly sussed that their opponents were vulnerable to corners and long balls into the box, knowing that they would either win a corner from a block or any Pompey hoof out would be going straight back to their players so they could build another attack.
Jackett's response was to make it even easier for Rotherham to win the ball in midfield by sending the small frame of Connor Ronan to play alongside Ben Close at the heart of the midfield, meaning there was nobody to tackle the Rotherham players as the onslaught continued.
Ollie Hawkins also replaced the ineffective Brett Pitman up front and yet wasn't able to win many long balls up to him because he was surrounded. On the odd occasion he did, he had no support as both Lowe and Matty Kennedy were almost playing as extra full backs and rarely got forward, so the lone striker was left isolated and may as well not have been playing.
It was frustrating to see a group of players who clearly have attacking talent play such a containing game and not react to what Rotherham were doing and that is mostly down to the manager. However, his players were also so careless in possession whether they were clearing the ball out of play or giving away silly free kicks to put themselves back under pressure.
This match simply highlighted further that Pompey miss a physical presence in centre midfield. Close hasn't shown that he is capable of mixing it up away from home but it's up to the manager to change the attitude of his players on the road and perhaps even take a look at his own decisions and what he can do to stop his team capitulating as it seems to be becoming a more regular occurrence.
In honesty, the only surprise was that the goal came so late in the game and had Pompey escaped the New York Stadium with a goalless draw then it would have barely been deserved. That's the third time this month that Pompey have conceded in the last few minutes so maybe it's down to fatigue given how a small squad has had to play a fair few matches in the last month.
The result means Rotherham have leapfrogged Portsmouth in the table and Peterborough have also overtaken them, meaning they are now ninth in League One and now four points off the play-offs.
It's still a good position to be in and there is still plenty of football to be played this season. But reinforcements are going to be required before another tough match at home to Shrewsbury on Saturday.
Pompey Player Ratings
Luke McGee: 7 - Coped with the pressure well.
Nathan Thompson: 7 - Another dependable showing.
Christian Burgess: 7 - Some crucial blocks.
Matt Clarke: 6 - Clearances were woeful at times, meaning the ball came straight back.
Sylvain Deslandes: 6 - Shaky start but grew into the game.
Ben Close: 5 - Lacking in every respect.
Dion Donahue: 7 - Seems to have put his poor start to his Pompey career behind him.
Jamal Lowe: 6 - Wasted one excellent opportunity but was Pompey's biggest threat on the day.
Kal Naismith: 5 - Barely involved after the break and still looks short of match sharpness.
Matty Kennedy: 6 - Did his defensive duties well.
Brett Pitman: 4 - Wasted the best chance of the game and was rightfully taken off for his lack of involvement in the match.
Substitutes
Alex Bass
Theo Widdrington
Adam May
Connor Ronan (Deslandes 66): 6 - Looked bright when he had the ball, which was about three times during the game. Clearly the wrong player to pick in the circumstances.
Kyle Bennett
Conor Chaplin
Ollie Hawkins (Pitman 66): 5 - Barely saw the ball and lost the aerial duel with the Rotherham defenders.