While driving on diesel I would like to use the regen function on the controller to charge the battery so after an hour or so to be able to use it again. The question is.., can the regen be used for prolonged periods of time and use the motor as an alternator or is there a limit?
As long as the load you're placing on the motor as a generator is no worse than the load it's designed to see as a motor, it should work fine, and that load is at the normal RPM the motor runs at for that amount of load as a motor.
If the motor as generator runs at slower RPM for the same load it would normally run as motor at higher RPM, it could get hotter than it should.
The controller's regen function you'd need to look at how the specific brand/model you're going to use is designed, and what kind of waste heat the function will generate.
There are multiple ways regen can work. Some of them are more efficient than others.
Some just passively rectify current from the motor phases back to the battery as long as the motor voltage is higher than the battery voltage. This is the cheapest in terms of power cost--the battery does nothing except receive charge current, and the controller does nothing but passively rectify phase to DC. But it creates heat in all the FETs because they are being used as diodes, and there is loss across those diode drops. The controller doesn't even have to be "on" to do this (doesn't even require it's "brain" to be functional).
Some actively switch the motor phases with the right timing to generate "flyback" pulses that are "always" higher voltage than battery so that regardless of motor RPM they'll charge the battery at least a little bit. This uses some power to keep running the controller, but has less or no diode losses in the FETs, so the controller can heat much less if it's driven correctly by the controller.
Some of them are not designed for power recovery (and technically aren't regen but are still often labelled as that), but instead are designed for higher braking power and actively use power to force the motor to push against the direction of rotation. You don't want those kinds of "regen" for your purpose.
(they also heat up both the motor and controller significantly)
Your battery must also be able to safely charge at the current your system will generate.
It might be safest to have a BMS of some type on the cells that monitors them for HVC and LVC, and uses an output line to tell the controller to shutdown if either one is tripped. (using a typical BMS to just disconnect the battery from the controller during regen can cause the controller voltage to spike and blow up the controller, depending on the type of regen used and the motor RPM at the time).
If the controller is advanced enough to monitor current and modulate it to not exceed the safe cell charging current, I'd definitely recommend setting it up that way.
