Hall Sensors'b'Gone

[amberwolf]

After a few hours I went back out, and it had cooled to ambient (which is now 90F).

Turned on battery connect, and switched on SFOC at LED Status panel switch, with no throttle and no phase and no serial connection, just battery power (14s NMC, 57.5v per the CA) and the LED Status panel.

Monitored temperature both by hand (touching the side of the case) and the strapped-on thermal sensor (next to the center heatsink bolt).

Within 15 seconds, it had gone up enough to feel it was warm, to 97F, and continued to rise. I shut power off at that point, not waiting for it to get to ridiculous temperatures, as it indicates there is a problem in the SFOC itself of some type, and is not caused by any attached motor or throttle (or temperature sensor).

While it is possible that damage has occured to the SFOC *from* a motor, that means it is now damaged in a way that causes it to rapidly heat up doing "nothing", it doesn't require that a motor be attached to cause the heating at this point.


I'm going to wait for it to cool down again, and then retest without switching the status panel on, and see what happens. (still not atached to anything but battery and panel)

Then I'm going to reflash to a diferent firmware, and retest (also without connecting anything but batteyr and panel), with and without switching the panel on.

 

[incememed]

That does seem like a developing fault in the power stage.

Disconnect the controller, make sure voltage batt pos and batt neg is zero, and measure the resistance over upper and lower part of each phase leg. (phase a, b, c to batt pos and batt neg).

Bummer, I was looking forward to having you test the new startup function.

 

[amberwolf]

Today, using the ambw20190415.hex file, the flash went ok. I didn't attempting to write any settings, so it's whatever the defaults are. Again leaving only the LED status panel and battery connected, I turned on the battery switch, and then the panel switch, monitoring temperature by the same external sensor as before.

It did not go up rapidly, but it did very slowly go up a couple of degrees over the next several minutes, however this could be because I took the controller off the trike outside where it's 97F, then did the flash in the house, where it's 20 degrees cooler, then took it back out and did the above test. So it would still be warming back up to ambient; it started at 95F and ended at 98F (one degree above ambient). That's within the error margin of the sensor readings.

I discontinued the test so I could bring it inside and do the resistance tests, but after lunch I'm going to take it back out and leave it on until either I'm sure there is or is not a notable temperature rise.

If there is no rise, I'll bring it back in and try uploading the correct system settings to it for the trike, and retry the test (again with only batery and status panel attached). If I can't upload settings on this fw (cant' recall if it's the 0413 or 0415 file that doesnt' allow upload) I'll update to the 0504 fw and redo the above tests, then upload settings to it, then keep on with these tests.

(I don't know why it would have a problem with the oct18 fw that it worked with before, and not have a problem with newer stuff, but I'll still systematically test all these things anyway; it may tell us something useful).


If no rise, I'll hook up the motor and retry, then if no rise the throttle, then retry, if no rise, then I'll use the throttle to test the motor offground.

Etc.


Resistance tests, controller disconnected from everything (not mounted on the trike), using a fluke 77-IIIa on autorange:
-- voltage batt pos and batt neg: 0.01v (even after shorting them together and releasing)
-- resistance over upper and lower part of each phase leg (meter pos on batt pos for those tests, meter neg on batt neg for those tests). Value starts at the below, and continues slowly rising over time:
== phase a to batt pos: 1.147 megohm
== phase a to batt neg: 1.175 megohm
== phase b to batt pos: 1.084 megohm
== phase b to batt neg: 1.165 megohm
== phase c to batt pos: 1.171 megohm
== phase c to batt neg: 1.196 megohm
Where phase a is arbitrarily the leftmost of the three coming out of the end of the SFOC5, with the SFOC's mounting tabs at the bottom, and the heatsink bolts at the top.

Additionally, a resistance reading from batt pos to batt neg starts at 3.935kohm and rapidly increases, over a few minutes to 1.8 megohm, then begins falling slowly. A crappy harbor freight meter testing the voltage across the fluke's output during the resistance reading gives 0.1v at 0.9megohm, and 0.9v at 0.715megohm, 0.06v at 0.47megohm, etc. Took around 10 minutes to fall from 1.8 to 0.47 megohm, stopped at that point to try the settings upload and other tests.

 

[amberwolf]

Couldn't get settings to upload on 0415 fw, so went to the 0504 fw, and retesting just that with default settings, only status panel and battery power attached, after 10 minutes (starting from ambient outdoor 99F temperature) had increased to 103F, just sitting there. But nowhere near the rapid heating it experienced with the Oct18 fw.

Is now cooling to ambient, then I will reprogram to the settings for the trike, and retest just that.

stats
12 34 7F FF 00 00 2D AA 31 E0 00 03 1A 89 1A 94 02 2A 02 4E 04 6D FB E8 04 13 00 00 00 00 1A 2E 00 0D 00 00
says internal temp only 35c, 97f. When I pulled it off teh trike (after writing this post, except this part edited in later), the external sensor said 107F, and it was noticeably warmer than the outside air (at 99-100F), but not particularly hot to my hand.



Pondering: Is it possible that the Oct18 fw installation did actually fail somehow, something corrupted, despite the positive verification by ds30loadergui at fw install, and *that* could cause this heating issue, along with the non-normal operation I saw that same day? I suppose simply redoing the oct18 fw install would tell me if that's the case. If the next few tests are normal, then I'll do that.

 

[amberwolf]

After 10 minutes just sitting with battery power on, and status LED panel switch on, using 0504 firmware with normal settings for the trike, nothing but battery and panel connected, temperature only reached 102F per external sensor.

Powered off at battery connector, and let cool to ambient (100F).

Hooked up throttle and phase wires to the 4504 motor on the left side. (trike still on it's side, wheels offground)

Powered on battery and status panel switch; let it sit for 20 minutes, only reached 103F. Note that at power on, the motor started to spin quickly for just an instant, perhaps a quarter turn, then it quickly decelerated to zero. No significant wiggling noted.

Slowly engaged throttle, motor spun up normally. Held throttle at full once I reached it, for several minutes, only saw another 3 degree rise in temperature on the external sensor. Controller felt warmer than that by hand, but not really hot, just very warm.
Continued for a total of 13 minutes of no load full throttle, with only one degree further rise in temperature.

Let it sit no throttle for another few minutes, temperature remained changing between 106 and 107F, probably from the breeze cooling the sensor itself.

So the heating problem definitely is affected by the firmware, though exactly what is causing it at it's root I don't know.

I guess the next test is to put the Oct18 fw back on there and see what happens, first with default settings, and then with the trike settings. May not happen tonight though.


Stats after bringing the controller back inside:
12 34 7F FF 00 00 3C BD 47 49 01 E2 1A 73 1A 99 1F 01 15 39 1D 06 20 E8 14 BF 15 62 70 FE 19 F9 07 D2 09 E7
indicate it started at 45C (113F) and reached 50C (122F) internally.

 

[amberwolf]

Question, because I can only find this reference (and where I asked about this here:
https://endless-sphere.com/forums/viewtopic.php?f=2&t=30680&p=1451972#p1451972
but didnt' get an answer, AFAICS):

https://endless-sphere.com/forums/viewtopic.php?p=1394460#p1394460

Absolutely, the way regen would be implemented in vector control is negative torque demand (instead of just zero, as throttle off would normally render), which would be revoked as the vehicle approaches zero rpm. The torque magnitude would be a setting, the negative demand would kick in when the conditions Throttle off and Above a certain rpm are both true.

As of now, by increasing the Torque down-ramp setting, power can certainly be regenerated while slowing down the vehicle, at the expense of coasting.

I don't get any slowdown of the wheel, offground, even in the most recent 0504 fw, with TDR at it's max of 5.000%. Based on the quote above, I would expect at that max to get the wheel to spindown very quickly offground, when the throttle is quickly changed from full throttle to zero throttle (or near-zero in case zero means the SFOC just stops doing anything, which I doubt). But that doesn't happen, and AFAICT it should.

In any previous onground testing I also don't get any slowing of the vehicle (but given the mass, perhaps it has to have a higher capability to notice, than just 5% max for the TDR? or would need actively negative torque? (not implemented yet AFAIK?)). ???


FWIW, I'd like to see it implemented like this:
https://endless-sphere.com/forums/viewtopic.php?f=2&t=30680&p=1450038#p1450038

The first is to set it up so if you just let the throttle snap back to zero, it ceases positive torque, but doesn't apply negative torque. Then, if throttle is resumed, it works normally. Positive torque if throttle "demand" is above present "need", negative torque if it throttle demand dips below present need. But if throttle resumption demand is below present need, it does not do anything (no negative or positive), *until* need drops below present demand or demand is raised above present need, in which case positive torque is applied, until again demand drops below need and negative torque is applied.

The "throttle off with no negative torque" roll-off time would be configurable, at least within a small range, so it could be adjusted for various users' reaction times and systems. If it's set to zero, the function is disabled, and it applies negative torque at any time demand is below need.

This would be intuitive and useful, though I don't know how hard it is to rewrite the firmware to do it.

Note that "simple regen" can be substituted for "negative torque" if regen is all that's available in the SFOC5.

 

[incememed]

Encouraging. Resistance values also look ok.

Note that at power on, the motor started to spin quickly for just an instant, perhaps a quarter turn, then it quickly decelerated to zero.

This has been fixed in subsequent fw-updates, will roll this out to you shortly.

Torque Down Ramp -- This does what, again?

Torque up and down ramp limits how fast a change in torque reference (throttle) is fed to the q-current (torque) regulator.

I would expect at that max to get the wheel to spindown very quickly offground, when the throttle is quickly changed from full throttle to zero throttle

The accelerating torque on the wheel is the sum (with sign) of retarding torque and applied torque, the retarding torque of the offground wheel being the bearings and air drag.

T_acc = T_ret(neg) + T_app(pos)

When applying full torque to the motionless offground wheel it accelerates almost instantly to full speed, so T_app>>T_ret. When dropping the torque reference to zero, one eventually gets T_app=0, and

T_acc = T_ret + 0

So, a very small retarding torque is acting on the wheel, and it takes forever to slow down. No matter how steep the ramp, T_acc is positive until the very last millisecond when T_app goes T_app<(abs)T_ret.

To spin down very quickly, T_app must be T_app<<0 (regen).

On the other hand, for the onground wheel being accelerated up a steep hill (i.e. add gravity, ground friction and vehicle air drag to T_ret), a sudden drop in torque reference has a greater impact.

Torque ramp was something I put in back when I was doing BLDC-control, for which I didn't have any regulator. The PI-regulator adds smoothness, as does the throttle signal filter, so maybe it's no longer needed. The up ramp is useful when inexperienced riders try out the bike, but the same can be achieved by dialing down the torque limit. Anyone riding the SFOC, feel free to feedback.

There will occasionally be a regen component, even as the torque reference is never set below 0. The active power going into the motor is at all times P = i_q x v_q. When either of these is negative and the other is not, current is regenerated back to the battery. So, when taking i_q from its full value down to zero, v_q can at times be negative (the more so the more aggressive the PI-loop coefficients are set). Also, when approaching zero torque (i_q=0), i_q may momentarily undershoot into negative territory before settling in.

Thanks for the suggestion, regen is on the list to be looked at.

 

[amberwolf]

This has been fixed in subsequent fw-updates, will roll this out to you shortly.

Ok. I should be able to continue testing this coming Wed, if not sooner.

In the coming weeks, I'm also replacing the generic trapezoidal controllers that normally run the trike's motors, with a pair of Grinfineon sine-sensored/trap-sensorless controllers, because at least they're identical so whatever I do with one the other should also do, and anything that's different means it's caused by the attached motor instead. To do this I'm making a new wiring harness that will "parallel" the existing one (which will continue to run all the old stuff, until the new stuff is completely ready to just swap over throttle input and motor phase/halls, and turn on). It'll be a bit complex, so I can switch things on or off or run both motors from one throttle, or each from it's own, or both or just either one from PAS, or even disable PAS, etc. Been planning on this for quite a while, but am still working it all out.

The SFOC stuff will remain separate, running off the old harness, so the new switches, buffers, etc., won't affect the SFOC testing.

Thanks for the suggestion, regen is on the list to be looked at.

I appreciate the complete explanation...but I guess I'm still a bit confused.

I had thought there was actually already regen (which would imply significant slowing of the wheel when applied even offground) based on this (copied from the quote of you in my last post):

As of now, by increasing the Torque down-ramp setting, power can certainly be regenerated while slowing down the vehicle, at the expense of coasting.

Another separate question:

Is there any "surplus" 5v current available from the SFOC, to run an external device? If so, how much, safely?

I ask because it's been suggested that whatever serial interface (BT, etc) module ends up being used over here:
https://endless-sphere.com/forums/viewtopic.php?f=30&t=95633&p=1472632#p1472406
be powered off the SFOC 5v. I expect that it will require a separate traction-battery-voltage-to-5v DC-DC module to run whatever serial module is used.

I'm trying to gather help to create a user interface for the SFOC: (because the present method of setting it up and getting ride stats is...shall we say, cumbersome at best, and there isn't any way at present to utilize the realtime serial data stream, other than logging it to a file and manually parsing it and painstakingly manually converting each hex value to human-readable data).

 

[incememed]

That will be fun. 20 mA is available, details on p. 32 of manual.

 

[j bjork]

I'm trying to gather help to create a user interface for the SFOC: (because the present method of setting it up and getting ride stats is...shall we say, cumbersome at best, and there isn't any way at present to utilize the realtime serial data stream, other than logging it to a file and manually parsing it and painstakingly manually converting each hex value to human-readable data).

I think it sounds like a very good idea :thumb:

A display would be good too
My computer skills is way too bad to be able to help with either one though

 

[amberwolf]

That will be fun. 20 mA is available, details on p. 32 of manual.

Thanks--I missed that. I'll post it over in the thread to see if ti's enough, or if a DC-DC from the traction pack has to be used.

I tried to figure out how to do it myself...but I just couldn't, so now I"m working on gathering help to do it.

It can't be that hard for people with experience doing this sort of thing (which unfortunately isn't me).

Right now, the suggestion is that the program itself be built as a web app, so it could be accessed from any device. But I guess this complicates the hardware part that interfaces with serial...and I don't know anything about the stuff suggested so far, so we'll see how this works out.

THe good news is that if htis works out, then it could be used (with a bit of firmware/app modification) with any device that outputs a serial data stream.

 

[amberwolf]

And I forgot to ask:

Can the SFOC be providing 5v power to the device that's waking it up to send it settings and/or read it's stats?

Or does the device have to be the one to provide that power?

This is assuming the battery power stays connected to the SFOC5.

 

[incememed]

Only when being powered up can it enter programming mode, and only if its RX pin is pulled high (i.e. another device is signalling it wants to communicate).

 

[amberwolf]

That's waht I thought. So whatever is doing the programming needs it's own 5v power source, and needs to provide that to the SFOC5 via the serial port signals.

 

[amberwolf]

This has been fixed in subsequent fw-updates, will roll this out to you shortly.

Did I miss an update? I haven't seen any emails with fw, or anything new (other than a stat sheet from 6-8) on the google drive page...wondering if an email disappeared into the ether.

(I haven't reinstalled the controller yet, was waiting to see if the new fw would be normal or if the controller would have the heating issue again).

 

[j bjork]

What temperatures are you getting?
I have never noticed any heating when not driving, but I have not really looked for it ither. When riding I seem to end up in the 85-110c range often. Once I got a 125c protection.

But I guess our driving is a bit different, mine is mostly offroad.

 

[amberwolf]

What temperatures are you getting?

Before this problem cropped up, the controller didnt' really get hot, and the motor would only get hot if I used high phase currents and startups from stop a lot. (my riding is all on flat roads, usually in traffic).

Posts starting from here:
https://endless-sphere.com/forums/viewtopic.php?f=2&t=30680&start=200#p1463758
have details on the heat problem, but temperatures so high that even many minutes after an extremely short test ride with less than a few seconds of intermittent low-power use of the motor, the SFOC itself was still over 130F. Minutes before that reading (with a remote IR sensor on the case) it was so hot I could not touch it...and the outer jacket of a cable passing the case of the SFOC was melted, so it was probably over 150F to start with, while the motor, wires, etc were all still at ambient, 77F.

Subsequent testing using an external sensor pressed against hte case (not nearly as quick to react, so probably never got to see the actual peaks things reached), saw at least 140F in just seconds of the controller not even running the motor under any load.


AFAICT, this is caused either by something strange in the reupload of the Oct18 fw or a problem in the flashing itself, etc. (despite "success" messages). It didn't appear to happen with a reflash of a different firmware.

Further testing hasnt' been done yet, was waiting for the new firmware Incememed mentioned.