Hmm...
You'd have to use a different relay (a DPDT, methinks), but otherwise, yeah, that'd work.
Some diodes would be needed to keep the caps from running the motor after it's going slow enough for the caps to overcome the BEMF from it, of course, but that's a simple matter. And same with the batteries, you'd need an LVC to protect the caps.
But I'm starting to like this idea. In addition to a voltage boost, you could also work out a relay array that allowed you to use the cap's voltage to get going from a start, saving a bit on the batteries.
Mebbe there will be a budget allotment for a small supercap bank in my future...