So...you want your drummer to, in essence, be the one to `drive the bus'?

Since his tempo will not be perfect, you're also going to need to be able to manipulate the playback speed of the backing track, in real time, correct?
Wouldn't it be easier for the drummer to let the sequencer and/or audio track drive the bus?