The most likely problem is a battery that is simply incapable of handling the current demand, for any of a number of reasons. The motor (or controller) wouldn't be the cause, unless it's simply that the battery isn't made to handle the power required to operate it, or the controller LVC is not set correctly for the battery being used (or the battery being used is not high enough voltage for the controller)..
Is power completely shutting off? If so, it's the BMS trying to protect the battery against damage that could lead to a fire.
Or is the controller simply stopping motor operation? If so, it's probably the controller LVC set too high for the battery being used, so when voltage drops below that, the controller stops responding to prevent overdischarging the battery it thinks it has (vs the one that's actualy there).
What are the complete battery specifications, for the original battery? A link to the sale page may show us helpful info.
What are the complete battery specifications, for the second battery? A link to the sale page may show us helpful info.
Note that if the only specs you have for the batteries are their voltage and capacity, we'd definitely need links to the sale pages to get the rest of the info from there. If those pages don't have that info, then it is safest to assume the batteries are incapable, and that you'd need to buy a battery capable of handling a minimum of whatever the current limit on the controller is set to.
What specific controller did you originally have, and how is it setup (programmed), if it has any settings? A link to the sale page may show us helpful info.
What specific controller do you now have, and how is it setup (programmed), if it has any settings? A link to the sale page may show us helpful info.
If the batteries are capable per their specs (and it is them rather than the controller shutting down) then they are defective and would need to be repaired to make them meet their specs and work correctly on your system. This may be possible depending on your experience and/or willingness to learn (and possibly break things; it can happen in troubleshooting).
If it's really 12kw, then at nominal 72v it's a battery current of 167A to do that. (less when it's full voltage). As voltage drops it would take more current to do the same work, if the controller is setup to do that, so it could be harder on the battery as it gets closer to empty. (150A at 72v is only about 10.8kw)
Do you have a wattmeter on the bike, to see what the actual Amps, Watts, and Voltage are? (seeing the peak Amps and Watts, and the lowest Voltage (Vmin) is often useful in cases like this, to know how much voltage sag there is at what level of current draw.)
If you don't have one you can get one pretty cheap, though they aren't waterproof so you may have to come up with a housing for it. I bought this one (but haven't tested it yet):
https://www.amazon.com/gp/product/B013PKYILS
it's only good to 100A but they make ones that go to 200A+ that would work for your system.
I use teh Cycle Analyst v3 from http://ebikes.ca every day on my SB Cruiser trike, but it isn't cheap (because it is a lot more than just a wattmeter...but it is a very good wattmeter, too).
Bikestick said:
This is a new build. 150a, 12kw,72v. After I received the battery it shuts down every ride mostly after it gets down to 75% or lower. I thought maybe a bad battery so and the battery company won't respond. So I bought another from a local store. 35ah. It shuts down also. so I bought a new controller and it still shuts down. The last time I pushed it home as always ,put a charger on it for 20 minutes, unplged it and it came back at 80% battery. Is it possible that there is some thing wrong with the hub motor that it is drawing to much energy and the BMS shuts down?