A123 AMP20 cell diagnostics

[arkmundi]

Thread seque..

It just so happens I work in warranty at A123 in Livonia. I can offer a couple of comments, though I know very little about how your BMS is intended to work.... Tearing the cells down would give you a reasonable idea of what the cells were actually exposed to, and thus give you a shot at understanding what actually happened here.....

Wow, a real live A123 person at ES. That's totally cool. I have an A123 AMP20 12S 36V pack, and extra cells to swap in should any fail. No BMS. But I have a batteryspace.com smart charger specific to a 36V lithium-ion pack. Its been doing beautifully charging my pack to around 42V and then shutting itself off. I've never seen an "over-charge" situation. And I monitor my usage so I'll never exceed a 75% discharge.

But every once in awhile, a cell fails and I replace it. So I've got these dead cells that I don't know what to do with. I don't know why a cell fails for me. I NEVER over-charge or over-discharge. It just hasn't happened. I've been that careful of my investment.

You have suggested that looking at a cell can tell you what may have happened. Is there a place I can send my dead cells? Someplace there's an expert? And then recycle the cells to base metals? That would be extra specially cool if that could happen. Or if not, a picture of the cell??

 

[arkmundi]

A picture of the cell would be a good start and a description of how the failure came to be discovered, etc.. You might not need to do any teardown to get to a satisfactory answer.
One important question: are these "grey market" cells you're working with? I haven't had a chance to look at your A123 pack thread in any detail yet.

dh

So, a picture? Looking at the dead cells it does not look any different from any other cell, on appearance. I have not torn one apart. Yes, my cells were sourced from Victpower, the "grey market" and not new from Cellman. What kind of picture do you want? Do I tear the aluminum foil off, exposing the electrochemical contents? More info would be helpful.

I discovered it because the pack "sags". I then do a voltage test of each cell individually and discover a dead cell, meaning voltage is near-zero. I attempt a single cell trickle charge to see if I can get back up, but not. They no longer hold a charge.

 

[wb9k]

I assume you're monitoring individual cell voltages and regularly verify that the pack is reasonably well balanced. A 42V HVC would put you at a max of 3.5V per cell, which is very safe/conservative as long as the pack remains balanced. You wouldn't have to get too far out of balance to exceed 3.6 here and there, and that's not good for any extended period of time (Li plating occurs, resulting in capacity loss and impedance growth).

I'll assume balance is OK (please tell me if you're not sure). I see you've posted that these cells came from Victpower. What do they look like when they have failed? Any discoloration around the edges? What do the tabs look like when you get them? I've seen some incredibly dumb practices from Victpower on another forum recently where they tried to weld new tabs onto cells that had the original tabs cut off all the way down to the pouch. Any electrolyte smell coming from any of these?

I would suggest putting some kind of cheap voltage monitor like a couple of Cell Log 8S's or the like and setting an audible low Voltage alarm at about 1.5 or 2 Volts. That a cell gets to near zero before you discover a problem tells me you may not be watching your cell Voltages closely enough. In a gradual failure (and the unspectacular ones almost all are), you should notice that the pack is getting more and more out of balance as capacity fades on the failing cell(s). Do you monitor individual cell voltages regularly, or just pack voltage?

dh

 

[arkmundi]

The cells from Victpower all looked fine coming in. Half tab, meaning they cut them out of a pack, probably part of the whole Fisker welding machine calibration fiasco, but that's just speculation. They all tested fine, meaning them came in around 3.3v, and would charge and discharge fine. Then they go into my pack.

Yes, I keep them in balance, using the manual approach. That means active monitoring of the individual cells. I have celllogs that I used for awhile but have more recently jettisoned in favor of active monitoring. Sometime I became convinced that using a smart charger so there's no over-charging combined with never exceeding a 50-75% discharge was a good approach to battery pack health.

I've never seen a voltage dip below around 3.2v. Except when suddenly its zero and won't hold a charge. The dead cells all look the same as when I put them in the pack, again visibly on the outside. No discoloration or exuding electrolyte or anything like that. No puffing. No nothing except the voltmeter telling me they're zero and won't hold a charge.

Again, many thanks if you can help with a diagnostic.

 

[wb9k]

What you're describing--a cell going from 3.2 to 0 Volts with no other noticeable event is hard to believe. There was almost surely some kind of thermal excursion if the Voltage really did fall that much in a matter of minutes, but I guess you could have missed that as it apparently was not too severe (if it did happen that way).

My first hunch would be that somewhere along the way the cell was overdischarged or driven negative, and then used some more. The beauty of grey market cells is that this can happen even before you get them. You really have no way of knowing. A cell that's been subjected to that can work for a while, then suddenly fail when an internal short develops. Cell teardown would show if this was the case, but is not something you want to undertake too casually.

Grey market stuff is of course dicey from the get-go. The cells you have could have been rejected for any number of reasons or exposed to who-knows what before being handed off to you. If you post a picture of what they look like when you get them, I can offer more insights on their potential history. Are they US or Korean cells?

It might also be helpful to know how the pack is assembled. I assume that's covered in that other thread of yours?

Going to bed soon....

dh

 

[arkmundi]

Thanks. They are "genuine made in the USA" A123 cells, confirmed by many, including myself. They're not counterfeit. They are what was described at the get go as "half tab", as the tabs are visibly cut. They are also pock-marked with holes, indicating they were once welded and have been pulled apart. You can see a picture of these cells on this thread

My battery pack has been performing to spec, so I have no reason to doubt their authenticity, albeit off the grey market. Its four cells I have had to replace, one at a time, not all together. The pack will deliver enough juice to get me home, but with a noticeable sag. Its why I say they fail suddenly. The pack charges, I go off on my ride, all is well and then, whoops, there's that sag. I go home and test each cell individually and what, there's a dead cell, as in nothing and it won't hold a charge.

I'd be more than happy to attach some pics. I can take a picture at each step in dissection, first with skin on, and then I can cut in. Always wanted to do it, but had no real reason until now. Just need you to step me through it.

 

[t3sla]

I don't believe he is questioning the authenticity of your cells, rather he'd like to trace the cells origin through the OEM chain.
A S/N batch can tell someone when they were made, who they were sold to etc
That means knowing what application they were used in, which gives you a greater understanding of your cells history.

Grey market has no rules, it's actually where rules go to die.

 

[agniusm]

Funny enough calling grey market cells and A grade cells. I thought all cells made at Livonia factory were "grey" from the start:

"Such flaws -- which were known to A123’s management but were concealed
from investors -- included: 1) a defective welding process that resulted in part from defective
welding tabs; 2) improperly coated electrode plates that were inserted into the prismatic cells,
causing additional damage at later points in the production process; 3) widespread
contamination of the Prismatic Battery parts due to egregious violations of quality control
protocols caused by employees removing parts and examining them by hand; and 4)
improper inclusion of parts classified as defective by the quality control team in the Prismatic
Batteries.
9. As a result of the defects in A123’s Prismatic Batteries caused by its
numerous manufacturing flaws, on November 4, 2011, the Company abruptly announced that
it was revising its revenue guidance downward by 22% due to “an unexpected reduction in
orders for battery packs from Fisker.” The Company, however, failed to alert investors that
this “unexpected reduction” of orders represented an estimated 3,000 units, which was
essentially A123’s entire battery production for Fisker for the fourth quarter of 2011. On this
- 4 - Case 1:12-cv-10591-RGS Document 34 Filed 08/31/12 Page 5 of 66
news, the Company’s shares plummeted from $3.53 to $3.17, or 10%, on unusually heavy
trading volume of 3.4 million shares.
10. Seeking to avoid a wider selloff, A123 downplayed Fisker’s order halt,
reassuring investors that the Fisker “reduction” only reflected a “temporary” inventory
reduction, and that its relationship with Fisker remained “strong.” However, A123 concealed
from investors that the halt was a result of substantial defects in the Company’s batteries,
which would ultimately lead to a massive recall of all of its Prismatic Batteries...
Full article here:
http://securities.stanford.edu/1048/AONE00_01/2012831_r01c_12CV10591.pdf
Basically AONE was knowingly selling defective cells when QC staff was told that all cells must PASS
My thoughts are that most of them will fail prior its time.

 

[wb9k]

Funny enough calling grey market cells and A grade cells. I thought all cells made at Livonia factory were "grey" from the start:

Nice threadjack. Yes, there have been plenty of problems in Livonia and elsewhere in the company, but the scope isn't nearly as severe as once believed. The Fisker recall was a big knee-jerk mistake. Defect rates on returned packs and modules was a small fraction of what was expected. We have made a ton of progress since then, and continue to do so at a more reasonable pace than a year or two ago when the company was trying to grow much too fast.

I see you use the cells, so you're probably a frustrated fan of the technology, a position I'm sympathetic to. That said, please don't hijack technical threads to put me on the spot for business decisions I had nothing to do with.

Thanks,
dh

 

[wb9k]

I don't believe he is questioning the authenticity of your cells, rather he'd like to trace the cells origin through the OEM chain.
A S/N batch can tell someone when they were made, who they were sold to etc
That means knowing what application they were used in, which gives you a greater understanding of your cells history.

Grey market has no rules, it's actually where rules go to die.

You are correct, sir.

So let's get started. I looked at the photo in the other thread. These appear to be cells that were built into modules which were later torn down and scrapped. Those holes in the anode tab are a fuse (we call it a zipper fuse), not damage from welding or anything accidental. None of the guys over on Prius Chat who were using these knew that, and I'm sensing the same here. I always point this out because I see a lot of cells around that don't have the fuse punched into the anode tab. These are cells that were rejected (or stolen) at the cell level, prior to being built into any module. I would avoid these, as I consider them most likely to have an interior defect, which may or may not be a big deal. They may have been rejected for marginal capacity with no further consequences in use, or they may fail very prematurely, or even spectacularly. The other thing to watch out for is "replacement tabs" welded into position, sometimes after peeling back a portion of the seal at the top of the pouch. This very often leads to poor isolation between the anode and the pouch, which in turn results in decrepitation of the Al pouch and electrolyte leakage, often in a matter of days. The tell-tale sign of this failure mode (welded tabs or not) is black spots around the perimiter of the cell that develop a corroded look and eventually become leakage sites. None of this is relevant to your situation it seems, but it's good background info.

In your case, I'm still curious how often you check the balance of your packs and your methodology. Please be specific. It's hard for me to imagine that everything really is just fine right up to that last ride--especially when you claim to never discharge more than 75%--though anything's possible I suppose.

Let's hold off on dissection for now and fill in some more gaps. Have you measured the DC or AC resistance of any of the failed cells? Is your pack ever allowed to sit unused for extended periods of time at low SOC? Are you replacing the same cell position in the pack over and over, or are the failure happening in different spots each time? That should be a good start...

dh

 

[Spacey]

I've just checked on one of my Victpower packs and the outside cell has puffed up. Pack has never been discharged and had the BMS balance lead disconnected when stored for 2 months.All cells charged initially.

Have pierced it then superglued the tiny pin prick hole I made just to see how it would react. Is holding a charge..... But I take it the cell is useless?

 

[wb9k]

I've just checked on one of my Victpower packs and the outside cell has puffed up. Pack has never been discharged and had the BMS balance lead disconnected when stored for 2 months.All cells charged initially.

Have pierced it then superglued the tiny pin prick hole I made just to see how it would react. Is holding a charge..... But I take it the cell is useless?

You would probably be surprised to see how long that battery would continue to operate in that kind of state, but yes, that thing is trashed. Sounds like it either overcharged or has an internal short. The electrolyte got hot enough to gas out at some point. Not sure how puffed up it was--some light puffing is to be expected (and one important reason to keep these cells under compression in a pack), but once you've got a hole in the pouch, that's it. It's a long slide downhill from there. Isolation faults can also develop through the path of leaking and dried electrolyte, depending on the construction of the pack.

dh

 

[etriker]

When cells are scrapped and sent to China do you discharge them first ?

 

[arkmundi]

In your case, I'm still curious how often you check the balance of your packs and your methodology. Please be specific. It's hard for me to imagine that everything really is just fine right up to that last ride--especially when you claim to never discharge more than 75%--though anything's possible I suppose.

For a while I was checking each cell after every ride. I do this with a simple voltmeter. Having confidence all is well, I checked less frequently, but no less frequent than once a week. My tell-tale in this equation is total pack voltage, which hoovers between 39.5v and 42.0v, after discharge and after full charge respectively. So I'm monitoring "normal" and if anything seems off, I check each cell's individual voltage. If its low (less than 3.2v) I'll charge just that cell with a single cell charger. That does not happen very often, however. Then on these occasions there's a spectacular fail.

The fail for all four cells that have failed is the same: all the other cells are OK - its just the one cell. And its voltage is zero and it won't hold a charge after trying to bring it back to life. I then strip the one cell out of the pack and replace it from my hoard. I originally bought 14 cells and then came along and split a carton of 14 with another ES'er, so 21 cells for a 12s pack plus the four that died. I have five cells left. I do not intend on buying any more at this time, so when I loose the five remaining cells, my pack is dead.

So that's the reason for my question here on this thread. If I can get to the real reason for cell failure, it may be something I can act on. Or if the real reason is the cells themselves, as variously speculated on, then there's nothing I can do and it is what it is. One suggestion is that I'm running without a BMS, so get one and put it on. But I have not as yet been convinced it would add anything to the equation. Indeed there is much discussion here of to BMS or not-BMS, and that having one may complicate things and may itself be a source of endless frustration.

Let's hold off on dissection for now and fill in some more gaps. Have you measured the DC or AC resistance of any of the failed cells?

No, but I'll do that and get some results back.

Is your pack ever allowed to sit unused for extended periods of time at low SOC?

No, I'm using it all the time, on a daily or every other day basis. My habit is to use and then when back home to charge it. So it sits overnight on a full charge.

Are you replacing the same cell position in the pack over and over, or are the failure happening in different spots each time? That should be a good start...

Each of the four cells has been in a different location in the pack. On my saga thread here, I speculate as to causes when the first cell died:

If you're into the A123 AMP20 pack line, I'd like your opinion on what went wrong.

A. The cell was bad from the beginning, just a casualty of the salvage that put these half-tab cells on the market through Victpower.
B. Without a BMS, and through usage, the cell became dangerously low and lost its ability to hold charge.
C. With the accident (hit by car and knoked to the pavement), the jar in fact caused damage to the pack - fortunately just the one cell.
D. With the bike in the shop for repair, and since I inadvertently left the controller on and Watts Up meter connected, the Watts Up drained the cell and killed it in the process.
E. An alien being that lives on nanophosphate li-ion fluff invaded and sucked it dry.
F. Something other

and the thread here:

What happenned for cell 6, happened again for cell 7 - just won't hold a charge. So I repeated the remove & replace operation with my last remaining extra. The pack is back to holding a full charge at 42.94 volts, an average 3.58V per cell. But the question remains what happened.

So with cell #1 being the positive pole, I lost #6 on 1Oct12, #7 on 2Oct12, #11 on 21Oct12, and lastly #12 (the bottom cell) on 20Mar13.

Again, many thanks!

 

[999zip999]

They pay to have them shipped to China for recycle. Which are sold around the world and back to USA plus shipping. Mine work fine 24s have replaced one with high IR. Use at 30 amps and 6 cycles at 60amps. There are many differents in these cells and can't be explained. Just try and match ah and don't over discharge and i always balance charge as bluck charging they get all over the place.

 

[etriker]

They pay to have them shipped to China for recycle. Which are sold around the world and back to USA plus shipping. Mine work fine 24s have replaced one with high IR. Use at 30 amps and 6 cycles at 60amps. There are many differents in these cells and can't be explained. Just try and match ah and don't over discharge and i always balance charge as bluck charging they get all over the place.

I bet the USA sends boat loads of good li ion cells of many types that are useful for ebike use to China to be recycled.

 

[bigmoose]

I always point this out because I see a lot of cells around that don't have the fuse punched into the anode tab. These are cells that were rejected (or stolen) at the cell level, prior to being built into any module. I would avoid these, as I consider them most likely to have an interior defect, which may or may not be a big deal.

So, just to be excruciating clear, in your experience ALL full tab cells that hit the market are either:

• Defective cells sold for scrap
• Good cells stolen somewhere in the process chain to Fiskar

Is this correct?

 

[arkmundi]

Wait, wait! This is a dialogue between me and wb9k regarding diagnostics on A123 AMP20 packs where cells have failed and nothing more, please. Please don't diver this important conversation. wb9k I have confidence can help me narrow the reasons why I've had four cells fail out of 16 (25%). If the two of us can discern the underlying reasons, I believe all ES'ers with these cells will benefit.

 

[wb9k]

When cells are scrapped and sent to China do you discharge them first ?

Let's be clear. The only recyclers I know of for our cells are in the US and Canada. A123 DOES NOT EVER send cells to China for scrapping or any other sort of disposition except for delivery of good product to its customers in China. This export is being done by third (or fourth) parties in direct violation of orders from A123. The company is aware of the vast grey market and regularly shuts down these sellers whenever possible. Sometimes cells are discharged prior to teardown of modules, but this often does not happen, especially when there have been large quantities of material to scrap. All material bought through unauthorized vendors like Victpower should be considered stolen and potentially defective, AFAIC. I'm not saying that to call anyone out of bust anyone's chops, but those are the facts.

BigMoose asked about cells with uncut and unpunched tabs. If buying these through authorized third-party vendors, they are surely fine, but I would not want to use the pouch cells without the zipper fuse punched into them. That fuse is there for a reason, but I'm not sure if we put that feature into raw cells that we sell. It really comes down to knowing your seller and the info in my first paragraph, I guess. Bear in mind that while Fisker was our biggest customer for a while there, they were never our only customer. The "recall" affected a handful of other customers as well.

Batteries being put into storage for any period of time should be set at about 50% SOC. Self-discharge and capacity losses are lower at 50% than 100%, but this still gives you a lot of headroom for avoiding overdischarge in storage. Always check SOC of all cells before using a pack after storage. Topping off and balancing is the best way to verify balance is still good.

I have a care and feeding guide for A123 cells that I can share, but the file is too large to post here, about 15 MB--and that's in pdf format. Any work around for that?

dh

 

[etriker]

To be clear.

A123 sends the recalled or bad scrap cells to US and Canada recyclers who sell them to China dealers who sell them back to US buyers ?

Where did they become stolen ?

 

[wb9k]

Having confidence all is well, I checked less frequently, but no less frequent than once a week. My tell-tale in this equation is total pack voltage, which hoovers between 39.5v and 42.0v, after discharge and after full charge respectively. So I'm monitoring "normal" and if anything seems off, I check each cell's individual voltage.

I don't think this is sufficient. You could potentially get way out of balance before you would see a problem looking at only the pack voltage.

If its low (less than 3.2v) I'll charge just that cell with a single cell charger. That does not happen very often, however.

How much lower than 3.2 have you seen? How often? Is it usually the same cell? Did you see this happen with any of your four failed cells prior to failure?

One suggestion is that I'm running without a BMS, so get one and put it on. But I have not as yet been convinced it would add anything to the equation. Indeed there is much discussion here of to BMS or not-BMS, and that having one may complicate things and may itself be a source of endless frustration.

I have several small packs with no BMS, and that's fine IF you're hyper-vigilant about managing the pack balance manually. I put sense harnesses on all my packs so I can quickly attach a Cell Log to verify balance at a glance. You should be checking this way at the end of every charge and discharge cycle--every other cycle at least--not just looking at the pack voltage, which will not change noticeably until things are already seriously awry. If you're well out of balance and don't know it, one good ride would be all it would take to drive a low cell negative and get a failure that acts just like what you describe. This seems like one strong possibility to me, and perhaps the only likely possibility that you would have any control over, so start there. There are some bad BMS's out there and those of course should be avoided. That anyone would say that you should never use a BMS with a Li battery pack is nutty, IMO. If your BMS gives you problems, improve it, don't just throw the whole concept of control out the window. That's just asking for trouble.

Please do provide cell resistance measurements. Very good on the charge and use schedule. We're getting closer....

dh

mod edit to fix syntax of quote panels. No change of content.

 

[wb9k]

To be clear.

A123 sends the recalled or bad scrap cells to US and Canada recyclers who sell them to China dealers who sell them back to US buyers ?

Where did they become stolen ?

I'm not sure what the exact path is, but most of our teardown work has been done by third parties who have perhaps not been babysat as well as they should have been. I suspect those parties are telling us they're recycling cells, but they're really selling them. I'm sure there's also some theft by individuals within A123 who are doing the same. I raid the scrap bins regularly myself, but I never, ever sell the stuff. Control over the scrap process has been less than optimal it would seem, but my involvement has been very small so I can't speak to exactly how sellers like Victpower are getting them.

dh

 

[etriker]

I am running a 11s 5p pack made with A123m1 cells made in 2006, recycled from Dewalt packs that is still going strong so I know ya'll make some awesome cells !

Thanks for joining and helping out !

 

[bigmoose]

I have a care and feeding guide for A123 cells that I can share, but the file is too large to post here, about 15 MB--and that's in pdf format. Any work around for that?

Perhaps print the pdf to two other pdf's, one with the first half of the document and the second with part 2? Perhaps that will get under our attachment size limit. The guide would be great information.

 

[ambroseliao]

I have a care and feeding guide for A123 cells that I can share, but the file is too large to post here, about 15 MB--and that's in pdf format. Any work around for that?

Get a free Dropbox account and then share your file from there.

https://www.dropbox.com/referrals/NTY2OTAyOQ?src=global9
Once you've created your account, upload the file to a folder in Dropbox and then right click on it and select: Share Dropbox Link... If you use the link, we both get 500MB additional space. It comes with 2GB already.

Paste the link into a message here, like this!
https://www.dropbox.com/s/22vmeqpy8f2ksop/a123_systems_amp20_data_sheet.pdf

Dropbox is a great file storage utility that you can get to from anywhere and on just about any device. PC, Mac, Android, iOS...

I'd love to see that document!
Thanks DH.