How many amps is the switch rated for? I did this on my trek and it also lowered my speed under load as the switch was unable to put out the amps under load.
Also a dead or dieing cell will do the same thing under no load it will go the same speed but once under load it will be your weakest...