VESC as gas generator, watchdog resets and unbalanced current

[Jopj]

Hi everyone, i'm looking for pointers on how to debug some annoying and apparently random issues plaguing this range extender I made with a weed whacker engine and a VESC. It uses software running on the scooters MCU talking to the VESC (a Flipsky mini fsesc6.7) to actuate the throttle valve via the vesc servo output, and vary the braking load.

Basically, everything works absolutely perfect for anywhere from several seconds to several hours, after which the VESC reboots. According to the terminal, this is nearly always a WATCHDOG_RESET error, though I have seen UNBALANCED_CURRENT a few times too.

What I have tried so far, with no change in behaviour:

• Try another VESC in case of broken HW
• Change control mode from FOC to BLDC
• Rise and lower switching frequency
• Make the control logic extremely sloppy to avoid fast transients
• Tweak flux linkage and observer gain
• Try different FW versions (6.02, 6.05)
• Try different motor (freerchobby 6384 190kv and a Maytech 6374 190kv)

What makes this confusing to me:

• Doesnt seem like the usual loss of tracking that causes ABS_OVERCURRENT
• Quite low powers and currents are involved, max ~700W or so, on a 12S pack
• Data looks fine, voltage is stable even with max power so should not be an obvious bad connection
• Cannot use triggered sampling on error as the mcu reboot

Here are some samples recorded from when it is running.

Engine running, no command for vesc:


Engine running while loaded. The slow pulses aren't because of motor control, but the reactions to the single cylinder engine's power stroke.


The biggest lead I have is the unbalanced currents thing. There does seem to be some bias to the phase currents, phase 2 getting higher positive values and phase 3 higher negative ones. Even in idle, they average 0 but each has a value other than that. But shouldnt any offset be calibrated out at start by the firmware? Or does it only zero out the summed current? And if this was a hardware fault, what are the chances of two different vescs having the exact same fault? In the BEMF plot the phases seem to be quite equal, on the other hand:



As some extra info, neither motor I tried has any issue passing detection, they work on this vesc with much higher currents without issues than used with the generator. I had modified the VESC firmware to include CAN messages for setting power limits, but I have switched back to the stock firmware to debug, though there is no difference in behaviour.

Does anyone have ideas how to figure out what's going wrong here?

 

[qwerkus]

Are you sure you included mcu yield points (sleep ticks) in every thread you added / changed on the vesc ? It's very easy to miss and can trigger the watchdog. Bombarding the vesc with input wether CAN or uart can also mess things up.
For motor fine tuning to figure out hidden errors, you might want to consider this post by mxlemming who is quite good at this stuff.

 

[mxlemming]

Watchdog reset means the MCU locked up and ceased computation. This cannot be solved by fiddling with motor control parameters.

Most likely it relates to the custom logic you have implemented, some state that causes a hard fault or such.

It can also represent the hardware being clobbered with emc or having issues.

You might also want to just wipe the whole chip and start again with a fresh install. Happening on multiple vescs implies either logic or wiring though. I'm not aware of this watchdog reset being a common occurrence for anyone else.

 

[Jopj]

Watchdog reset means the MCU locked up and ceased computation. This cannot be solved by fiddling with motor control parameters.

Most likely it relates to the custom logic you have implemented, some state that causes a hard fault or such.

It can also represent the hardware being clobbered with emc or having issues.

You might also want to just wipe the whole chip and start again with a fresh install. Happening on multiple vescs implies either logic or wiring though. I'm not aware of this watchdog reset being a common occurrence for anyone else.

I have flashed stock firmware which reboots the exact same way, so it should not be those new messages I added. Thanks for the guide by the way, its existence here was the reason I figured to post to this forum in the first place!

Going to keep it stock until I figure this out just to reduce variables. The engine's igniter is quite close to the VESC and certainly a source of emc, maybe I can try mounting the vesc far away from it. Apart from that, the CAN and battery wires do have some tens of centimeters length to them but that can hardly be helped. Can is twisted pair too for whatever it's worth. Will be harder to do anything about those.

 

[Jopj]

Are you sure you included mcu yield points (sleep ticks) in every thread you added / changed on the vesc ? It's very easy to miss and can trigger the watchdog. Bombarding the vesc with input wether CAN or uart can also mess things up.
For motor fine tuning to figure out hidden errors, you might want to consider this post by mxlemming who is quite good at this stuff.

My mcu is an external thing that's talking to the vesc via can, it shouldn't mess with any internal goings-on of the (stock firmware running) generator VESC. It sends a single message at 50 hz using canbus speed of 500k, which doesnt cause issues with the other vesc (that controls the drive motor) on the bus running same firmware and sending same telemetry with the same rates. I feel quite confident that this shouldn't be an issue.

 

[TRS300]

My mcu is an external thing that's talking to the vesc via can, it shouldn't mess with any internal goings-on of the (stock firmware running) generator VESC. It sends a single message at 50 hz using canbus speed of 500k, which doesnt cause issues with the other vesc (that controls the drive motor) on the bus running same firmware and sending same telemetry with the same rates. I feel quite confident that this shouldn't be an issue.

I would start changing conditions to see if it impacts the VESC program from locking up. If you do not run the engine but leave all the communications running does it still lock up?
I'm not exactly sure where the VESC sits in this design so it's hard to make suggestions other than start deactivating things to see if it stops the problem.

Can bus communication should never cause a problem like this but it's hard to say how it's set up in the VESC. What MCU you using to communicate with the VESC? I'd be curious to see that code if you wouldn't mind sharing it.

One other thought is that the can bus communication in itself is not causing problems but if you are sending the VESC commands via can bus, the vesc might be locking up while trying to execute the command. Meaning what you are asking it to do might cause the vesc program to lock up long enough for the watchdog to activate.

 

[Jopj]

I would start changing conditions to see if it impacts the VESC program from locking up. If you do not run the engine but leave all the communications running does it still lock up?
I'm not exactly sure where the VESC sits in this design so it's hard to make suggestions other than start deactivating things to see if it stops the problem.

Can bus communication should never cause a problem like this but it's hard to say how it's set up in the VESC. What MCU you using to communicate with the VESC? I'd be curious to see that code if you wouldn't mind sharing it.

One other thought is that the can bus communication in itself is not causing problems but if you are sending the VESC commands via can bus, the vesc might be locking up while trying to execute the command. Meaning what you are asking it to do might cause the vesc program to lock up long enough for the watchdog to activate.

If engine is not running, there haven't been any lockups. Likewise, when the engine IS running but the current command to vesc is zero, there hasn't been any lockups either. Interestingly, when the vesc is cranking the engine (about 30A phase current) there haven't been freezes yet, but could be just chance as it takes very little time. So far it hangs only with a negative current command.


This is where the vesc sits logically speaking, as the "generator VESC". It's only connections to the rest of the system are the power cables, as well as the CAN wires. Only CANH and CANL are connected. I'm using a Teensy 3.2 as the controller, with a MCP2562 can tranciever. The code for that is found here in all it's ugliness.

• EMC from engine ignition - move vesc farther
  • Engine runnig by itself never causes problems though, and its ignition is the same no matter what VESC does
• VESC 5V Voltage drop from throttle servo movement - power servo with other source?
  • Unlikely, quick servo movements make it no more likely than static servo, servo is lightly loaded
• "Unbalanced currents"
  • Whether or not it causes this problem, surely it's not right?

The last one is what I understand the least. If the VESC is idle, surely there should be around 0A measured on each phase? Currently, seems like each phase has up to 5A measured current when idle, but these sum up to 0A total. Seems like there's something wrong here.

 

[TRS300]

Appreciate the link to the custom MCU code. At 1800 lines of code, I'm not going to try and figure out what it's doing. Can you give a brief summary as to what it's doing. Specifically, I'm interested in what exactly are you looking at that drives the generator output / ICE engine load?

 

[scianiac]

The last one is what I understand the least. If the VESC is idle, surely there should be around 0A measured on each phase? Currently, seems like each phase has up to 5A measured current when idle, but these sum up to 0A total. Seems like there's something wrong here.

Could be just noise, Flipsky is not known for their low noise controllers. Maybe check the current offsets? From my understanding it auto calibrated the current offsets at boot which is generally good although there are some conditions where it goes wrong, you can set them and then have it not calibrated them at boot as well.

 

[Jopj]

Appreciate the link to the custom MCU code. At 1800 lines of code, I'm not going to try and figure out what it's doing. Can you give a brief summary as to what it's doing. Specifically, I'm interested in what exactly are you looking at that drives the generator output / ICE engine load?

While I try to debug this, it's just "throttle servo position from throttle lever ADC, braking current = (rpm - target rpm)*tuning_value". Basically just a lazy P controller.

Normally, in the manual transmission emulation mode the throttle lever control throttle servo, and it's using the VESC PID modified to do braking only where target engine RPM is traction motor RPM * virtual gear ratio. It then governs the traction motor power to equal the power so generated. In CVT mode, virtual gear ratio varies smoothly based on throttle lever position instead. In generator mode it opens up the throttle and sets the target RPM to the engines max power RPM, while the traction motor is directly controlled by the throttle lever.

 

[Jopj]

Could be just noise, Flipsky is not known for their low noise controllers. Maybe check the current offsets? From my understanding it auto calibrated the current offsets at boot which is generally good although there are some conditions where it goes wrong, you can set them and then have it not calibrated them at boot as well.

Could be, and I will check this next. It is quite solid for noise though, looks more like bias. I'll definitely try manually tweaking those as soon as I figure out how.

 

[TRS300]

"The biggest lead I have is the unbalanced currents thing. There does seem to be some bias to the phase currents, phase 2 getting higher positive values and phase 3 higher negative ones. Even in idle, they average 0 but each has a value other than that. But shouldnt any offset be calibrated... "

After giving this more thought I asked myself..." what is unique to this situation that the ESC would not normally see? " And the answer was, "a power stroke". The vesc is outputting power generated by the motor attached to an engine. That power output generates a load force on the engine yet the power output delivered by the engine fluctuates a great deal during a stroke cycle. In fact during the compression stroke the engine is actually consuming energy, not delivering it. The ESC would never see power input like this normally and the decreasing power output from the engine during the power stroke, moving to no power output at all during the compression stroke results in one half of the revolution of a two stroke motor not providing any power to the generator/ESC on every rotation. Meanwhile the esc is trying to manage this erratic power input from the generator which it's simply not designed to manage.
If the motor was on a 4 to 1 gear ratio and not a 1 to 1 ratio it might help, but maybe not.
I can easily see how you would imbalances in the phase output on a single cylinder 2 cycle engine input.
Another option would be to introduce a weighted flywheel to smooth out the power from the engine using the kenitic energy from the flywheel mass.

 

[Jopj]

"The biggest lead I have is the unbalanced currents thing. There does seem to be some bias to the phase currents, phase 2 getting higher positive values and phase 3 higher negative ones. Even in idle, they average 0 but each has a value other than that. But shouldnt any offset be calibrated... "

After giving this more thought I asked myself..." what is unique to this situation that the ESC would not normally see? " And the answer was, "a power stroke". The vesc is outputting power generated by the motor attached to an engine. That power output generates a load force on the engine yet the power output delivered by the engine fluctuates a great deal during a stroke cycle. In fact during the compression stroke the engine is actually consuming energy, not delivering it. The ESC would never see power input like this normally and the decreasing power output from the engine during the power stroke, moving to no power output at all during the compression stroke results in one half of the revolution of a two stroke motor not providing any power to the generator/ESC on every rotation. Meanwhile the esc is trying to manage this erratic power input from the generator which it's simply not designed to manage.
If the motor was on a 4 to 1 gear ratio and not a 1 to 1 ratio it might help, but maybe not.
I can easily see how you would imbalances in the phase output on a single cylinder 2 cycle engine input.
Another option would be to introduce a weighted flywheel to smooth out the power from the engine using the kenitic energy from the flywheel mass.

I thought about that too, but I'm inclined to ignore it via the following reasoning:

A power stroke is at maximum one half of a rotation, with the generator being a direct drive 7 pole pair motor, there should be ~3.5 full commutation cycles throughout it, meaning the "bias", if caused by it, should change that many times per power cycle.

This is in fact a 4-stroke engine, a GX25 clone. If it happened that the power stroke caused such a sharp peak that it'd only impact one part of the commutation cycle (which it does not, looking at the graph), 7 is not divisible by 4, so which particular phase would be out of line should change for every power stroke. That doesn't happen, it's always the same one.

Finally, when the engine is idling and the VESC is not driving the generator, the power cycle should not have any particular effect other than the speed change, as the vesc is not reacting to it. This is indeed so, but the bias remains, now solid and without the current pulses.

 

[TRS300]

Good points on the Commutation rate. However, one half of the rotation of the engine still is not providing a constant power force. I would try to see if a weighted flywheel helps smooth out the power delivery from the engine. It's one of the reasons why engines have weighted flywheels and why they are more important with fewer cylinders if the power output needs to be more constant. A weed whacker doesn't need smooth power delivery to do it's job. I worked my career in the energy and power industry. You'd never see a generator on a single piston engine. How could the power output ever be regular? And if you look back in history all early single piston drive systems always had huge flywheels (for a reason). The more cylinders you have the less need for a flywheel. Anyway, just my thoughts.

 

[Squilt]

Awesome project. Sorry I dont have any advice, just questions. I'm planning on doing a very similar build in the future. How did you go about selecting your engine, generator motor, and esc? I went into a rabbit hole a while ago trying to figure out what would be the best voltage for the generator to output relative to the battery voltage. Do you use regen on your traction motor? If so how do you keep charge current from getting too high while the generator is running?

 

[Jopj]

Awesome project. Sorry I dont have any advice, just questions. I'm planning on doing a very similar build in the future. How did you go about selecting your engine, generator motor, and esc? I went into a rabbit hole a while ago trying to figure out what would be the best voltage for the generator to output relative to the battery voltage. Do you use regen on your traction motor? If so how do you keep charge current from getting too high while the generator is running?

Thanks! I selected the engine based on three things:

1. Four-stroke. I don't want to be mixing in oil, and the wasteful nature of two-strokes offends me on a philosophical level
2. Small. It powers a scooter that's got more than enough electric grunt, this needs to be just enough for cruising
3. Cheap. It was a silly spur of the moment project so dont want to invest in some fancy twin cylinder aero engine

Turns out there are 4-stroke engines for hand-held tools such as trimmers, a common one was the GX25 and it's clones. That fit the bill, as it's good for about 600-700 watts, which it does actually deliver. Mine was about 90€ off Alibaba with shipping.

Selecting the motor used as generator was done based on RPM. The engine revs up to 8k RPM. I wanted to use direct drive for simplicity, and since I had a 12S LiPo battery, I'd need a minimum kV of 8000/42 = ~190. This is to make sure that even with the battery at low SoC, max RPM from the engine would not exceed the controllable range. I happened to have a 193kV motor at hand, so chose that. Vesc does up to 0.95 duty cycle, meaning this is actually marginal at the very lowest SoC, but I don't suffer problems as exceeding the duty cycle limited motor speed causes a strong braking force, at a time when the engine has very little torque at the top of its RPM range, and with the motor overpowering the engine to such an extent that it easily "wins out".

As long as the generator RPM stays within the VESC's controllable range, there's no need to worry about its output voltage, the vesc does that for you. You can just command the desired current, or as in my case, desired RPM, which results in whatever amount of power being produced at a given throttle opening being the generated power.

I use regen on the traction motor liberally, it is the main source of braking even when pulling the solar trailer. The charge current is not a problem for me as the scooter has 20Ah pouch cells that can easily soak up those peaks even with the generator running at full power. But if it were a problem, it wouldn't be hard to program the generator control to ease up whenever braking power reaches some certain level. I have set the vesc regen limit for the generator a bit lower than the max SoC voltage to deal with the battery getting full. Currently the generator control doesnt understand that this happens and it will start hitting the software rev limiter. Until I implement a fix, I just command the generator to turn off when that corner case happens.

 

[Tony01]

Watching. Interesting thread. Looks difficult. Have a dream to build a performance hybrid with drivetrain like a Prius with two motors acting as a CVT or “power split device” as they call it.

 

[Jopj]

I have ruled out the servo movements causing 5V drops as a cause, even with no load on the generator vesc 5V the behaviour remains the same. Next I will try moving things farther from the ignition, and testing yet another vesc, this time an older two-shunt flipsky. Oddly, fiddling with the current offsets causes no change in the current readings, but increasing them too far still throws an unbalanced currents error.

 

[Jopj]

So this appears to be solved, but in an unsatisfying way. I tried with an older 2-shunt mini fsesc, mounted far away from the ignition. No issues. Works right next to the ignition as well. So it looks like the 3-shunt mini fsesc is just somehow allergic to this application.

 

[mxlemming]

So this appears to be solved, but in an unsatisfying way. I tried with an older 2-shunt mini fsesc, mounted far away from the ignition. No issues. Works right next to the ignition as well. So it looks like the 3-shunt mini fsesc is just somehow allergic to this application.

Sorry I missed that you were using the flipsky 6.7 in your original post... Several people have had problems with these on the vesc discord. It looks in principle like a great piece of hardware but it seems they just don't work properly. Your solution is therefore annoyingly probably the only solution...