Aaron Mokoena headed home a corner in the 120th minute of this third round replay to save Avram Grant's men.
Pompey had been trailing to a fine strike from Leon Best in the first half before skipper Stephen Wright put through his own net in 89th minute to send the game into the 30 extra minutes.
Coventry City boss Chris Coleman had decided to rest Freddy Eastwood and play Clinton Morrison up front with Best and it proved to be a wise decision.
The Sky Blues took the lead in the 22nd minute when Best produced a stunning goal which was totally out of context with the rest of the game.
City keeper Keiren Westwood booted a long kick forward and when Michael McIndoe and Morrison won headers, the ball fell to Best.
The striker, being watched by several Premier League clubs, didn't both with any control he just spun around and lashed the ball in past Asmir Begovic.
It was a goal that Coventry just about deserved on the balance of play but there was not much to choose between the teams on a bitterly cold evening.
Every time Coventry went long from Westwood, the Pompey defence seemed to freeze.
Grant brought on John Utaka at the interval and the Nigeria forward gave the visitors a lift.
He produced on fine and run and cross which was intercepted by Richard Wood at the vital moment.
He then powered a volley wide and sent a header just past a post.
Coventry changed their formation to hold off the Premier League side in the second half but they were cruelly denied.
Portsmouth took a quick free-kick and when ball was flicked goalwards, Wright stuck out his head and diverted the ball in off the bar.
Neither side could really lift themselves in the first part of extra time.
Frederic Piquionne did get the ball in the net after eight minutes but he was clearly in an offside position and his joy was short-lived.
Everyone was preparing for penalties when, in the very final minute, Mokoena emerged unmarked in the area to head home and bring Portsmouth their first ever win over Coventry in the FA Cup.