It looked as though both sides would have to settle for a point after United had a late goal disallowed and Arsenal had a penalty overturned, but Rice sent the home fans into delirium as he opened his account with a deflected shot in the 96th minute.
Jesus then wrapped it up with a sublime finish after a counter-attack with United having committed everyone forward.
“I think I controlled it with my chin and just had a shot, if you don’t shoot you don’t score,” Rice, who arrived from West Ham United, told Sky Sports.
United were left crestfallen especially as they had what they thought was a late winner scored by substitute Alejandro Garnacho ruled out for offside after a VAR video check.
Marcus Rashford gave United the lead against the run of play in the 27th minute, lashing home a shot after a counter-attack, but their joy did not last long though as Martin Odegaard fired Arsenal’s equaliser less than a minute later.
Arsenal were awarded a penalty midway through the second period when Kai Havertz went down in the area but referee Anthony Taylor changed his mind after checking the VAR monitor.
Mikel Arteta’s side have 10 points along with three other clubsafter four games, two behind leaders Manchester City.
United are in 11th place on six points after already losing twice. “Everything went against us but I thought the performance was alright,” United manager Erik ten Hag said, adding that he disagreed with the decision to disallow Garnacho’s goal.
“I didn’t see that as offside but it is what is. It was such small margins today.”