Pompey had tabled a £4.5million offer to try and bring the midfielder back to Fratton Park but the Teeside club have rejected the offer.
"We made what we thought was a good offer for Gary but unfortunately it was declined and we will not be increasing our offer, Pompey Executive Chairman Peter Storie told the club's official website.
"He would have been a good addition to the squad."
O'nel left Fratton Park for the Riverside in August 2007.