I don't like the idea of challenges.
Managers could use them to simply disrupt the tempo of the game.
Put an official in a booth with multiple camera angles. He can look at fouls and determine if contact was actually made or a dive has occurred. All he has to do is buzz the referee and let him know what actually happened. Play has already been stopped, so it wouldn't be much more of a delay.
There could also be officials charged with watching the goal to see when and if the ball crosses the line, just like in hockey. Another thought would be to add two more linesmen. In American football, there are seven officials on the field in addition to the replay official. In basketball, there are three officials covering a fraction of the playing surface that football officials do. Same with hockey, except they cover a bit more surface...on skates. Four linesmen covering a quarter of the field would probably help get their eyes in the right positions more of the time.
The games have simply gotten too fast for three officials to adequately cover the entire pitch. Use more eyes and technology to get it right.
"Purgatory's kind of like the in-betweeny one. You weren't really shit, but you weren't all that great either. Like Tottenham."
I'll try being nicer...if you'll try being smarter.