Sounds very much as if the BMS is faulty and is partially shutting off. It has an electronic power switch internally that is supposed to turn off the power when either too high a current is drawn or if any cells in the battery pack discharge to too low a level. This should be a hard switch, that turns either on or off, but in your case it seems that the BMS is presenting a high resistance path, hence the varying voltage (which is just the capacitors in the controller charging up slowly).
The hard part is deciding what to do now. The BMS isn't one I recognise (someone else here might, though) and guiding you through diagnosing and repairing the BMS isn't going to be that easy. It might be worth starting a new thread with BMS in the title, together with some photos of it, in the hope that someone else on here recognises the type and either knows how to repair it or where to get a replacement.