Phoenix HV Analysis - Blown ESC observations

[swbluto]

So, I received a damaged Phoenix HV-110 controller from Recumpence and I did ask for one with blown capacitors to see what kind of problems are inherent with ESCs with blown capacitors. Upon receiving the controller, I didn't see blown capacitors but I did see two 1000uF capacitors on the ESC and it looks like stock HV-110s have four capacitors, 180 uf each, so I take it the blown capacitors were already replaced... I guess?

So upon opening it up, I did a little testing and found that there were heating problems with one of the MOSFET drivers, and it was obviously from drawing way too much current; as far as I can tell, no other parts on the control board are malfunctioning. The MOSFET driver's data sheet is at http://www.intersil.com/data/fn/fn9077.pdf and it's the SOIC-8 or 8-pin design. It appears the MCU still works as designed, the other two mosfet drivers are working and the switching supply still works. So, would anyone have any ideas why a mosfet driver would be overheating/destroyed? I'm thinking this suggests the supply voltage was exceeded at one time or something else, but what would cause that? If it's possible it was destroyed by a blown mosfet, it'd be nice to know that in advance because I want to try to replace the driver, but that would be kind of pointless if an unknown number of mosfets are blown because accessing the mosfets for testing is a little difficult with the "interlocking metal cyclinder" that's been soldered between the mosfet boards (Plus the top layer seems to have a metal plate glued to it). Btw, I tested the control board separated from the power boards, so the mosfet wasn't attached to the driver at the time of testing (It still malfunctioned when the power board was connected, I just coudln't access the drivers when the boards connected for temperature testing).

The board doesn't have any visible charring or other signs of "smoke" or other fairly visible signs of mosfets blowing up... but I wouldn't know if recumpence cleaned it up or not.

Does anyone know where I can get a replacement driver at mouser.com? Digikey has it, but I don't like shopping from them due to big errors on their part in the past and their high prices (It'd cost me like $11 for a mosfet driver from digikey with shipping included. ). Here's the product page at mouser: http://www.mouser.com/ProductDetail/Intersil/ISL6700IB/?qs=sGAEpiMZZMvQcoNRkxSQkoiKq6HKb18BipNRXhTSlZ0%3d . They have the 12 pin qfn package, but not the soic-8 that I need.

 

[liveforphysics]

If its getting hot, its likely ok, and its hot because its trying to switch fets that have internally sorted the gates to the drain of source.

 

[swbluto]

If its getting hot, its likely ok, and its hot because its trying to switch fets that have internally sorted the gates to the drain of source.

But it still gets hot from current over-consumption when it's disconnected from the fets... (the control board is separated from the mosfet board stack.)

Maybe it was damaged at one time from damaged mosfets, and it's failure mode was to consume too much current.

I guess I'll have to figure out how to test the gates to source and drain without taking apart the power board stack. Prima facie, it appears the gates are interlinked between each mosfet in a given row, and the sources and drains are connected.

Oh, wait, to test for a drain-source short, all I have to do is measure the resistance of the positive and negative leads. That will be easy. Now, I wonder... do gate-source or gate-drain shorts ever occur without a source-drain short? If not and if there's not a source-drain short, then I shouldn't worry about measuring the gate for shorts.

 

[swbluto]

I measured everything. The pos with each phase shorted between 4-5 mOhm while 2 of the black-battery/phase-wire combinations showed 40-50 kOhm. One combination showed 28.3 ohms, and this was the black-phase to black-battery, so I take it the black phase's low side mosfets have shorted. I wonder why I don't see any charring?

So I have bad fets, crap, crap, crap.

Does anyone know how to take the mosfet stack apart? It appears the heat-sinks are glued to the mosfets or something, and they don't readily "pull apart". It doesn't appear to have any mechnical fastening, suggesting glue.

Btw, a side comment on the heatsinking of the fets that I observed. In the HV110, there's like 6 mosfets that don't have a heatsink over them. I'm not sure why they're not worthy of heatsinking, but apparently this design works...

I also looked up the data-sheet and saw that thermal resistance of the mosfet's junction to case was 25 C/W, and that's probably close to these mosfet's junction to air, since the sink's and paste's thermal resistance is probably minimal. Well, anyways, it's no wonder these things overheat when some people are pulling a lot of phase amps. In my simulations, worst case "steady state" values for controller heating was like 100 watts. It seems that the RC controllers deal with this heating through massive quantities of fets. With 18 mosfets per side, and 6 sides, the hv110 has 108 mosfets. If the controller heating was 100 watts, that suggests an average per mosfet wattage of 1 watt. Assuming it was 25C/W, that'd be 25C above ambient which is what people seem to observe. However, I think some people are instantaneously seeing 800 watts of heating at extremely high phase amps during acceleration and at 8 watts per mosfet, that's like 200C above ambient. Clearly, that's enough to kill a mosfet... especially at lower motor RPMs, where the phase period is longer (Where only 36 fets experience that amount of heating for a longer period of time). Now if phase amps were sufficiently limited, I think one of these controllers has a fighting chance of surviving like my other one has.

 

[boostjuice]

Oh, wait, to test for a drain-source short, all I have to do is measure the resistance of the positive and negative leads. That will be easy. Now, I wonder... do gate-source or gate-drain shorts ever occur without a source-drain short? If not and if there's not a source-drain short, then I shouldn't worry about measuring the gate for shorts.

You do understand that MosFETs are a "normally on" device right? That is to say when charge is applied to their gate, a field effect is created which 'pinches' over the junction of the Source to Drain path, decreasing continuity, and turning the device "off" (high impedance). Without the controller being powered up, shorted phase sets should measure as the parallel equivelant of each FET's RDSon which in this case i would expect to be less than <0.001ohm for a working phase set.

Also, FETs can fail open circuit OR short circuit. However it's usually Gate>Source short and Drain>Source short.

There is a big diagnosis problem with having so many FETs in parallel if some have failed whilst others are good. The difference in parallel resistance of a set of good FETs, and say a phase in which half of them are in an open circuit failure, is bugger all. Likewise, if some or all have failed close circuit for a phase, then you will of course achieve a very low resistance reading that reads the same as a good phase stage based on the resolution limitation of a standard multimeter. dv/di is the only potential way to see differences without an expensive milliohm meter, and even then you couldn't be sure they're ALL good.
Testing each FET in isolation out of circuit is the only way to be totally sure of each FETs condition, and this is totally impractible for so many SMD packages. I bet Castle doesnt even bother trying to fix the power boards for this very reason. Attempting to salvage the brain/driver board is likely the most they bother with.

EDIT: Also, Gate>Source and Gate>Drain can be shorted without Drain>Source being shorted due to the gate oxide blowing

 

[swbluto]

Oh, wait, to test for a drain-source short, all I have to do is measure the resistance of the positive and negative leads. That will be easy. Now, I wonder... do gate-source or gate-drain shorts ever occur without a source-drain short? If not and if there's not a source-drain short, then I shouldn't worry about measuring the gate for shorts.

You do understand that MosFETs are a "normally on" device right? That is to say when charge is applied to their gate, a field effect is created which 'pinches' over the junction of the Source to Drain path, decreasing continuity, and turning the device "off" (high impedance).

What???

Maybe that's true for P-channel mosfets, but N-channel mosfets are normally off. They "turn on" when V_gs > V_th, with decreasing resistance for higher V_gs values.

Anyways, even assuming that I have not a clue what I'm saying, I do know that....

Battery red to...
phase red = 4 M-ohm
phase black = 4.3 M-Ohm
phase white 4.9 M-Ohm

Battery black to
phase red = 40.1 k-ohms
phase white = 40.3 k-ohms
phase black = 28.3 ohms.

One of them is not like the other...

 

[swbluto]

Testing each FET in isolation out of circuit is the only way to be totally sure of each FETs condition, and this is totally impractible for so many SMD packages.

'Tis true, which is why I'd probably just replace the whole faulty branch given that the "faulty" one isn't visibly obvious - wait, what am I saying? It should be easy, just short out the branch with a voltage supply and measure the temperature - the faulty one will be significantly hotter than the others.

BUT... I need to separate the mosfet stacks first and remove the heatsink(s)... any ideas? I'm thinking the heat transfer epoxy could be softened by heating it up, but it appears the most sensitive components to heat, the mosfets, are rated to 150c. I'm not sure if 250-275 degrees fahrenheit would work to separate it? It appears most thermal epoxies are rated to 200 degrees c, however... Well, actually, it appears the weakest link that couldn't be easily removed would be the plastic board interconnects so I couldn't use oven heating.... hmmm

 

[boostjuice]

Right you are...Brain Fart... I've been designing circuits with P-channel units lately.

 

[swbluto]

So, I ripped off the top layer of metal that the control board rests on and two mosfets had their top plastic part of their case ripped off. Looking inside... cool rainbow like colors. Also, one of the mosfets no longer had its legs attached to the inside of the casing, so I'm thinking that one is likely destroyed. So, I'll just need to desolder that one.

However, I tried melting the solder on the metal pipes that bonds the mosfet stacks together and that didn't work. It appears that the thermal resistance of that part of the ESC is far too low and it sucks away heat too quickly for my iron. It also appears that this is silver solder that they use; if it was lead, this wouldn't be a problem. So, I can't take apart the esc anyways...

So, in conclusion... if you blew your capacitors and the Phoenix HV ESC won't come on but the LED is dimly lit (You have to look closely), this is a sign that one of the mosfet drivers has been destroyed and is sucking too much current from the switching power supply circuit such that the voltage supply isn't high enough for the MCU to operate correctly, and also there's likely a destroyed mosfet. Unless you have some super ninja like soldering irons, the board stacks are nearly impossible to take apart so self-repair might be futile.

Also, as quick over-view of how the Phoenix HV ESC works... it uses a switching power supply, a silabs microcontroller, and it uses comparators for BEMF sensing, and uses three dedciated mosfet drivers with the link in the top post. The gate drive has 10 ohm resistors, and the HV110 uses 108 of these mosfets: http://www.fairchildsemi.com/ds/FD/FDS5672.pdf. It uses board interconnects to relay gate drive currents to the mosfet stacks. In other words, it seems to be pretty much a "text book" controller to me.

According to my brief pricing, it looks like the going rate for these mosfets is 60 cents per mosfets in quantities of 10,000 (Who knows if they get a further price break?). At 108 mosfets for the HV110, that's about $65 in mosfets.

 

[recumpence]

You can use a Dremel with a cutoff wheel to carefully cut each FET board apart from the other (the interlocking + - posts). Once apart, you can more easily remove the posts. Then resolder them together using a small brass tube to reconnect them.

Yes, that is silver solder. It is a pain to melt with a normal iron.

As an item of interest, that ESC was blown by running about 30% throttle up a long hill pulling two of my kids in the trailer. It was pulling 2kw at 30% throttle for about 20 seconds. When I reached the top of the hill, it just stopped running. No magic smoke.

I knew it would blow doing that. I had a large number of miles on that ESC and I wanted to see if it would take that abuse.

Matt

 

[swbluto]

You can use a Dremel with a cutoff wheel to carefully cut each FET board apart from the other (the interlocking + - posts). Once apart, you can more easily remove the posts. Then resolder them together using a small brass tube to reconnect them.

Yes, that is silver solder. It is a pain to melt with a normal iron.

As an item of interest, that ESC was blown by running about 30% throttle up a long hill pulling two of my kids in the trailer. It was pulling 2kw at 30% throttle for about 20 seconds. When I reached the top of the hill, it just stopped running. No magic smoke.

I knew it would blow doing that. I had a large number of miles on that ESC and I wanted to see if it would take that abuse.

Matt

Cool, will do!

I'm thinking that the ESC could be made operational just by replacing the driver and removing the offending mosfet. So, I can't wait to test out my theory! I'm still looking for a suitable SOIC-8 mosfet driver from mouser. It has to have the same pin-out and lower than or equal to rise-times, fall-times and propagation times. I get the feeling I'm missing something, though... (Like maybe there's a unique feature to this particular driver.)

EDIT:

Okay, so I used a dremel cutoff wheel to split apart the interlinking cylinders and then applied heat to the heatsink using a hot air gun, and then pried apart. No mosfets damaged this time, but I'm getting the hint that's because it was simply heatsink paste instead of thermal epoxy on this layer. So, I tested the layer and found that it contained the offending mosfet, and then used my temperature probe to test each mosfet and found the offender. I removed it, retested the board, and no visible short! Now, I also tested to see if the two previously damaged mosfets belonged to the same "branch" of mosfets and they did... so now I have 15 or 16 operational mosfets out of 18 on this black-phase low-side branch. I wonder if I should ameliorate that? Because, I don't know know what kind of solder that they're using but it doesn't seem to readily melt at 800 degrees fahrenheit , so replacement might be a little difficult.

I also tested the leakage current on the other power boards to test for any more source-drainage shorts. It seems to show a leakage current of 30 mA at 8 volts V_DS and the resistance seems to measure at 280 ohms instead of 4-5 M-ohm for the other phases... So, I pumped and the voltage to 15 volts and *snap*, the current jumped to the supply's limits. I wonder if other phases would show this behavior, or if there was truly still a shorted mosfet on this phase? Well, lucky me... the damaged mosfet should be easy to spot this time. (Now, crap... I'm wondering if I might have inadvertantly been pumping voltage through the intrinsic diode? I probably shouldn't do that. *Goes to check fet diagrams*)

I tested another phase to see if this may have been a damaged mosfet's fault or I may have betrayed an inherent characteristic of body diodes, and it didn't blow (I wisely decided to limit the current to much less than 3A on the power supply this time. ). So, there's likely still another damaged mosfet to remove.

 

[recumpence]

It sure would be cool if you could figure it all out.

Hey, what would it take to reverse engineer the FET setup and up the gate driver to drive larger FETs? I understand the startup issues with sensorless. However, I also like the lack of timing issues (like sensored setups have) and the low temperatures these things run at. I would love to use Castle's driver and software board to run a bigger FET board. Heck, I would even be willing to donate a HV160 (once they are out of backorder status) to use as a test bed.

Matt

 

[swbluto]

It sure would be cool if you could figure it all out.

Hey, what would it take to reverse engineer the FET setup and up the gate driver to drive larger FETs?

It shouldn't be hard. I'm pretty sure the gate driver that goes to certain pins on the interconnect, and the BEMF which can be traced from the quad-comparator can be found rather easily. The FET stage looks pretty simple, and there's six 10 ohm resistors (Presumably one for each gate), three 4.7 uH inductors and three small capacitors, likely .1 uF(that could be measured more exactly.), per power board; I'm sure the FET stage would be easy to figure out by comparison to known diagrams or from an existing knowledgeable member. It'd be tougher by directly measuring connections, though doable.

I'd worry about the added traces inductances of custom FET stages and the affects on performance. For sure, you probably don't want long wires leading from the control board and hooking up to your FET stage. I'm guessing that a better FET driver wouldn't necessarily be better, because trace inductances are the likely limiting factor. Although, if you could find a FET driver that had short protection or something, that could be a welcome upgrade. But, good luck in finding a SOIC-8 driver with the same pinout, with better characteristics. It's like finding a needle in a haystack, and you're not even sure if the needle exists.

 

[recumpence]

It sure would be cool if you could figure it all out.

Hey, what would it take to reverse engineer the FET setup and up the gate driver to drive larger FETs?

It's like finding a needle in a haystack, and you're not even sure if the needle exists.

Yes, but we are all welt aware of the HUGE haystack (lack of a heavy duty controller) that exists.

This should be a very worthy cause to persue.

Matt

 

[swbluto]

Okay, so I removed the other bad mosfet and I measured the resistances again. It turns out that the "resistance" is supposed to be 4-5 M-ohm as judging from the other phases.. but, the two separate phases that I had removed a bad mosfet from had a resistance of 40 kOhm and 144 kOhm. That sounds significant, but not as good as M-ohm... I wondered if the other mosfets in those phases had "slight damage", during use on matt's bike or if I accidentally damaged the surrounding mosfets when I was removing the bad one (Possibly too much heat)? Anyways, I'm guessing it's not a big deal but I wonder if it will be...

Oh well, only one way to find out!

Let's reassemble it once I get the new driver in, take it out for a spin and see if something blows!

I guess the proper thing to do would be to subject it to high V_DS voltage and see if it survives, but I don't have a voltage supply with that high of a voltage. (And if I use my batteries, that's going to be some serious smoke...)

 

[swbluto]

Btw, the mosfet driver is on the control board. If you wanted to use your own special driver, I'm guessing you'd have to connect the mosfet's driver input signal to whatever driver you wanted off the control board, but I'd be wary of that approach unless you carefully designed your own board and interconnect for minimal inductance (Although, the control signal should not have significant problems with trace inductance). Each driver drives 18 different mosfets at a time on the HV-110, and each mosfet has a gate+output capacitance of around 2600 pf. That means the existing driver should be suitable for a total FET capacitance of 46800 pF. A 4110 has a capacitance of around 9000 pf, which means the existing drivers would probably be suitable upto driving 5*6 = thirty 4110 mosfets.

BTW, did Luke already have a mosfet-driver+mosfet-board up and running? I might be able to connect a phoenix control board to his driver+mosfet board and see what happens. (Or he could try it out)

Well, on second thought, that would probably involve a lot of complications and unnecessary troubles. He'd probably be best to do it.

 

[swbluto]

So, I received the driver and installed it and tried it out.

The good news is that the control board is now at least getting power and the electric characteristics (Amperage) seem to be within spec for normal operation and the controller arms like normal. The bad news is that it doesn't appear to be either activating or sensing one of the phases, because it's now stuttering as if a phase was "disconnected". The fact the ESC is telling me "The motor is jammed" with one LED blink heavily suggests that it's not sensing it correctly. Why wouldn't it not be sensing it correctly? Heck if I know. I'm pretty sure the BEMF signals go to the quad comparator, and the quad comparator's output probably goes to digital lines on the MCU.

So, now I'm pondering what to do next. I'm thinking about scoping the comparator's input signals but I have no clue what I would be looking for. Maybe I'll scope the inputs and outputs and see what's happening. I guess if I'm seeing "appropriate"(at least somewhat similar) inputs but not an adequate output, it might be time to replace the quad comparator. But, you know what, tapping the quad's inputs would be hell - those pins are tiny!



Alternatively, maybe my driver isn't working. It'd be hard to believe since I'm sure it's soldered correctly and everything, but maybe. Maybe the input TO the driver isn't working! Now that'd be a conundrum because that would imply a pin on the MCU is fried, and unless I get a replacement MCU from castle creations directly, that's not getting fixed. I'm fairly confident the power board is working, though I'm sure I'll test that down the road if the other possibilities don't bear fruit.

 

[swbluto]

Upon further thinking, the MCU might have a burnt out pin because it's that ... that drove the mosfet driver. But, I would think it would be good practice to put a resistor on the MCU output pins, so I'm kind of doubting that. I think I'm just going to test all the transistors because that's probably the other thing that could be busted... maybe, and those are pretty simple to test, I think.

 

[mwkeefer]

sw, maybe time to send to castle for 60 or 90.00 replacement -

-Mike

 

[swbluto]

sw, maybe time to send to castle for 60 or 90.00 replacement -

-Mike

If minimizing my time/cost investment was my goal, I would've returned it at the very beginning.

But, no, I *have* to fix until it's known that I absolutely *can't* fix it!

Which, btw, is coming close. I was measuring the resistance on the input pins to the mosfet driver to the pins on the MCU and it turns out it was directly connected! No protection resistors or anything! Cheap bastards! :lol:

I still have to "prove it" to myself before I cease my attempts, but it's looking like the MCU pin might be burnt out given it has no protection to the mosfet driver's input and the mosfet driver, which was consuming large amounts of current, probably had exorbitant input current. If that's the case, I can't fix it. Only if I could somehow remap the internal logic of the driving pin to a different pin somehow, but I don't see how that's possible.

 

[mwkeefer]

While it is traditionally "good practice" to put resistors on the inputs for some current limiting and isolation of sorts, I do believe the MCU has internal pull ups which they are counting on to reduce the parts count, weight and such.

That said - I wonder if they even realize where their common modes of failure are, your right in that on anything critical i'd have matching 1% resistors as bias and protection - to me the gate drive and BEMF sensing inputs are "CRITICAL" so yea you would expect them to be there (then it would be a resistor to be repaired).

I'm all about the R&D aspects of this, if we could get these a bit more bullet proof - well all the better, but in my talks with CC they aren't real open to (or don't seem) input from our community as of yet.. Not in terms of over their predominant market.

What kills me though is a controller will allow itself to self destruct the FETs at partial throttle with PWM losses, that just plum seems contrary! And these supposed continuous ratings are at 5mph cooling airflow over the heatsink / controller... eh.

Keep up the great work, when I blow my HV... I'll send it your way for discetion.

-Mike

 

[liveforphysics]

While it is traditionally "good practice" to put resistors on the inputs for some current limiting and isolation of sorts, I do believe the MCU has internal pull ups which they are counting on to reduce the parts count, weight and such.

They are actaully critically important to save the FETs from the inductive bounce that can drive the FET back into it's transition zone again, which means loads of extra FET heat. Contrary to popular belief, it's not about protecting the driver circuit at all.

That said - I wonder if they even realize where their common modes of failure are,

The failure mode is pretty simple. They use surface mount FETs in a power application. They try to work off the idea that if you group enough FETs together, the resistance will be so low when it's cold, your battery will run out before they go into thermal run-away. This works fine for RC stuff, and intermitent applications.

What kills me though is a controller will allow itself to self destruct the FETs at partial throttle with PWM losses, that just plum seems contrary! And these supposed continuous ratings are at 5mph cooling airflow over the heatsink / controller... eh.
-Mike

The controller has no way to know, and no way to protect itself if it did know. It's fate is sealed when you choose the motor and the voltage. Choose a big turn-count motor wound with little wire, like a 10t astro or something, you're golden. Choose a motor like an 80-100 HXT, now you've got more load than 10 of the Astro 10t motors, and it's just a matter of time before smoke makes it's exit. The most extreme case would be the colossus motor, which if I'm not mistaken, they said the RC controller blew up just trying to start it spinning under no-load. LOL! Make sense too, since it was getting hit with >1,000amps phase current just running the pre-programmed starting sequence. lol

 

[swbluto]

While it is traditionally "good practice" to put resistors on the inputs for some current limiting and isolation of sorts, I do believe the MCU has internal pull ups which they are counting on to reduce the parts count, weight and such.

The MCU's data sheet is at http://www.keil.com/dd/docs/datashts/silabs/c8051f32x.pdf . When a pin is used as an output, the MCU designer tries to make sure the voltage output is above V_OH at all times with the specified current rating, and a resistor doesn't work well with that. But, if you're an end-user of the MCU, you can have the knowledge to know which output pins "deserve" a resistor. So, yeah, they probably have a pull-up when configured as input, but not likely as output.

Looking at the datasheet, I'm not noticing any mention of output protection but I did notice an absolute max output rating of 100 mA on a given output pin on page 29.

Looking at the datasheet, I notice it makes mention of "viewing and modifying" memory. I *really* doubt that castle creations would've left the MCU open to viewing the memory, but it'd be interesting to see what might be possible. Like, if I could access the program counter and then I tracked the resistance of the output pin to Vdd, I might be able to precisely find the line where the pin is set to output and then insert a different line. But I have my doubts viewing the program counter is possible ... or is it?

 

[mwkeefer]

It seems to use a 2C (2wire) interface for boundary scan and block read/write just like everything else (basically same as doing bounds scans on Jtag port).. It would be nice if you could dump the FLASH memory, we could probably hard edit the sucker to redirect that pin output making a functional unit again (worth a look right?)..

I'll review the datasheet later and see if I can't figure out a way to scan the flash and eeprom memory right off the chip - as far as CC locking it down, I have another purpose for 4 of their controllers but it will require being able to adapt the PPM input to I2C bus (faster, less latency, etc) quadracopter style I have a call in to tech support on this already to see if they will support it.

As a side benefit, if we can convert from PPM at 50hz base (1-2.5ms pw) then we can get much faster response time to throttle position changes and somthing like an overcurrent or temperature cutout system (which totally disables the throttle) could react in just a few ms as opposed to 20ms +

-Mike