CGPT >> KB

[E-HP]

I've been playing with ChatGPT for a few months. It's interesting. For ebike related info, its hit or miss for providing the correct response in one iteration. I've found that it will provide different or additional information if you just tell it it's wrong. It will even refine it's answer if you just say you think it's wrong. You can tell if it's answer appear to be making certain assumptions, which appear to be incorrect, and it will dig deeper, or provide the assumptions, and if you tell it you believe the assumptions are wrong it will try to make corrections and provide a better answer. (It's also sensitive to name calling, lol; on another topic, I told it that it was racist, and it made a big deal about the ways the response might be construed as racist, but defending it and provide explanations or make adjustments.).

Anyway, on some topics I've had to call them on bad assumptions or errors almost up to a dozen times, but in the end the response if very well formatted, clear and logical. For me, this is nice, because you can give it questions, etc., without the grammar, spelling, punctuations, etc. do not need to be perfect, but the end product is clean and can be easily copied and pasted. This is way easier than typing it all out myself, and the process to get ChatGPT to provide a correct and well formatted answer isn't too hard.

I was thinking to try out populating one or two of the stubs in the KB with ChatGPT generated, but user reviewed, responses, but would want others to review to see if the response is acceptable, as an experiment, before going further. I think it's one way to get the bots to work in our favor, because I can tell in some cases where the information came from ES.

On one occasion, after pages of back and forth before a clean/correct response was given, I complimented it, and it humbly accepted the praise, but then complemented me for pointing out it's errors and bad assumptions along the way, lol.

 

[neptronix]

You've already understood the problem with AI:
If you don't know the topic, it sounds super smart.
If you know the topic, you find yourself correcting it a lot or nudging it in certain ways.

Most new AI users arrive on 'mount stupid' extremely quickly. This is an analogy for The Dunning-Kruger effect.



Once a user understands AI limitations and knows it's strengths and weaknesses, it does become useful.
A person skilled in the topic's job is now to be prompter, fact checker, and grammar editor.

I've found this to be true in systems analysis and programming. But it took a while.
With writing documentation, i find that 'cross breeding' with other articles as context for style, a little pile of information on the thing, and an outline... after a human final pass, the results are better than what i'd write.

If we allow people to use AI to write knowledgebase articles, it's important to know what part of the curve they are on.



I think if someone were to demonstrate they knew how to wield AI to generate high quality information, they could get a stamp of approval from me.

I would love to build some kind of context retrieval system that would synthesize data from ES for you and make your job even easier, but we don't have the AI hardware to do it yet.

Conditions:
- human inspired and fully human checked, to prevent slop
- not too wordy like AI output tends to be
- same quality as an average human written article
- it lacks robotic tone, unnecessary lists, emojis, nonsensicality, errors, wrong emphasis, and other signs of "nasty looking spatter on it's welds", which indicates a human either directed it well, and/or did the appropriate cleanup work.

 

[E-HP]

You've already understood the problem with AI:
If you don't know the topic, it sounds super smart.
If you know the topic, you find yourself correcting it a lot or nudging it in certain ways.

Most new AI users arrive on 'mount stupid' extremely quickly. This is an analogy for The Dunning-Kruger effect.

View attachment 389555

Once a user understands AI limitations and knows it's strengths and weaknesses, it does become useful.
A person skilled in the topic's job is now to be prompter, fact checker, and grammar editor.

I've found this to be true in systems analysis and programming. But it took a while.
With writing documentation, i find that 'cross breeding' with other articles as context for style, a little pile of information on the thing, and an outline... after a human final pass, the results are better than what i'd write.

If we allow people to use AI to write knowledgebase articles, it's important to know what part of the curve they are on.

View attachment 389556

I think if someone were to demonstrate they knew how to wield AI to generate high quality information, they could get a stamp of approval from me.

I would love to build some kind of context retrieval system that would synthesize data from ES for you and make your job even easier, but we don't have the AI hardware to do it yet.

Conditions:
- human inspired and fully human checked, to prevent slop
- not too wordy like AI output tends to be
- same quality as an average human written article
- it lacks robotic tone, unnecessary lists, emojis, nonsensicality, errors, wrong emphasis, and other signs of "nasty looking spatter on it's welds", which indicates a human either directed it well, and/or did the appropriate cleanup work.

People might be able to get past mount stupid, simply by asking are you sure? or that doesn’t seem right. I think that forces it to look at how much of the response was built on assumptions, and look further into them.

I think it’s a smart writing aid for now, and you can see from the response on ebike stuff, where you need to correct an assumption, and correcting the assumption corrects the response.

I decided to see how it works on a topic I have little interest typing up, ebike lighting. I asked “how to add lighting to an existing ebike. The initial answer wasn’t bad, and it provided adding battery/rechargeable lights, e-bikes with controllers with a light output, and DIY with adding a buck converter. For the latter, it provided a rudimentary diagram of the components and where they are in the circuit relative to the other components (battery, switches, buck, lights). Also mentioned lights that work at full ebike voltage with built in drivers as an option, removing the buck from the diagram. Then I asked about using a specific factory ebike (Lectric), where the controller was being upgraded, and how to utilize the existing lights. That was OK but needed more work, and it had a hard time with a functioning brake light switched via the brake cutoff switches. That will take more work to get it to write what I want out of it. After 4 back and forths, the response looked somewhat acceptable.

I’ll try filling out a stub, and mark it CGPT assisted until it’s fully vetted.

PS. I asked it about some stuff about finding a local restaurant that had disappeared (it was actually a pop up that does business in an existing bar or establishment). It took a while to get good info from it (checking licenses and county record, etc.) but the end response made sense. I asked the exact same question a couple of weeks later, using a different device (without a ChatGPT login), and the response was the cleaner response I’d gotten near the end of the other session, so it incorporated what it “learned” from the earlier session.

 

[neptronix]

People might be able to get past mount stupid, simply by asking are you sure? or that doesn’t seem right. I think that forces it to look at how much of the response was built on assumptions, and look further into them.

What if they don't know enough to ask that and it's still wrong?
This has happened to me, had many arguments with the clanker. Luckily i know what i'm doing, otherwise i'm shippin' slop

I decided to see how it works on a topic I have little interest typing up... ...That was OK but needed more work, and it had a hard time with a functioning brake light switched via the brake cutoff switches. That will take more work to get it to write what I want out of it. After 4 back and forths, the response looked somewhat acceptable.

Yep.
For good results you want to come up with the outline yourself and also set the tone.
For best results, you also include some example human-created documents as the tone reference. This is how i get AI to write documentation like me ( i just need to do a few documents by hand to my best effort first )

I’ll try filling out a stub, and mark it CGPT assisted until it’s fully vetted.

Cool
Right now there is a little bug where it's picking up the funding notice on the page, don't mind it, we'll fix that.


If you want to set a draft state, set the visibility to 'editor'

PS. I asked it about some stuff about finding a local restaurant that had disappeared (it was actually a pop up that does business in an existing bar or establishment). It took a while to get good info from it (checking licenses and county record, etc.) but the end response made sense. I asked the exact same question a couple of weeks later, using a different device (without a ChatGPT login), and the response was the cleaner response I’d gotten near the end of the other session, so it incorporated what it “learned” from the earlier session.

I can ask different models the same question and get different answers depending on how the question is worded.
That's because there is a randomization function ahead of the text generation.

See it a lot with programming where i'm asking something obscure but it sometimes knows what i'm talking about, other times not. Luck of the draw.

Some paid versions of online services do offer a 'memory' and what you are describing works fairly well from what i hear.

 

[lnanek]

People might be able to get past mount stupid, simply by asking are you sure? or that doesn’t seem right. I think that forces it to look at how much of the response was built on assumptions, and look further into them.

Unfortunately, most AI offerings are just large language models trained on public data to produce the next most likely text sequence given a prior one.

It's not that it goes and looks into assumptions when it is wrong. It just tries to predict the text of someone who did that. None of the training data that was used in the beginning changed. None of the model generation that locked down how the variables generated from the training data relate to each other changed. It's just auto-completing a longer prior text sequence than before.

People who don't know the answer already would likely throw it off as much as help it.

 

[E-HP]

Unfortunately, most AI offerings are just large language models trained on public data to produce the next most likely text sequence given a prior one.

It's not that it goes and looks into assumptions when it is wrong. It just tries to predict the text of someone who did that. None of the training data that was used in the beginning changed. None of the model generation that locked down how the variables generated from the training data relate to each other changed. It's just auto-completing a longer prior text sequence than before.

People who don't know the answer already would likely throw it off as much as help it.

I’m just adding it to my workflow as a more efficient way of typing. I can ask questions, comment, etc. in very few words, one finger typing on my tablet and eventually get the result I want to paste in. A lot less typing so I don’t have to drag out my laptop.
We’ll see if the results pass the smell test on review.

 

[E-HP]

I was going to start a new thread, but after some typing, it sort of circled back to this thread, in a way.

My general observation is that there are an increasing number of battery related posts lately, but how they are being supported feels different. Different enough that I find that I'm reluctant to contribute my 2 cents to a discussion that requires the other 98 cents to result in a safe outcome. I try to stay clear of them, even if I can answer the direct question. My initial thought had been to create a new subforum for battery troubleshooting, separate from the Battery Technology subforum, which tends to discuss more general topics, rather than troubleshooting. The thought was to put the troubleshooting threads in one area where the ES battery experts could field them.

I decided to ask ChatGPT a few questions:
"Which members of Endless Sphere have the most extensive battery knowledge?"
"Which members of Endless Sphere have the most extensive battery knowledge and provide the most support?"

The first question resulted in a half dozen names that we all recognize, and provided supporting details for it's assessment, with medium high confidence.
The second question resulted in a breakdown of historical heavyweights and their contributions to specific areas within the subject, and the "modern" specialists covering general, to more specific support.

Probably not a surprise that amberwolf landed at the top of the heap when it comes to support on this subject, but not because of specific expertise. I asked why, and what about his support landed him at the top, and the answer was the missing component that has reduced my contributions on the subject.

The missing component is that he never leads with a solution, and is "less focused on the specific fix". Instead, he identifies and manages the risks and unknowns, before recommending any action. It then went on to describe 7 specific areas covered in order to get the full picture. This is important, as the current crop of newbie questions are usually prefaced by I know X or I know Y as though it's acquired knowledge rather than what they read from a Reddit post or Google Ai response, and it takes a question or two to sort that out, rather than just taking it at face value and provide a quick response.

So, I now realize that I've been leveraging the fact that a couple of key ES members have always filled in the gaps, allowing me to add my two cents, if I had anything to contribute at all. I could rely on Chalo to come in and just say how stupid the idea is then provide some choice reasons why, or amberwolf to step in and with a lot of typing, providing enough information for the OP to decide for themselves that it's stupid. Either way, house fire averted, and from my standpoint, either method, the carrot or the stick, is better than just answering the question.

Anyway, I have no solutions for the current situation. From the KB perspective, ChatGPT does a decent job summarizing what amberwolf's support has looked like with respect to the areas that need to be clarified in order to provide a full picture before offering a response, so maybe a pre-battery troubleshooting checklist might be something to add to the KB that might help remaining members provide support. I'd hate to read an article where someone claims that their house burned down because of advice they received here.

I asked several other questions, in different way, when it comes to minimizing risks when providing battery advice, and all of them pointed back to the secret sauce that amberwolf has always incorporated into his support, and what is missing today. It summarized the full chat with:
"The highest-value battery expert isn't necessarily the person who can name the correct BMS or cell fastest. It's the person who can get from an ambiguous symptom to a defensible solution without creating a more dangerous condition along the way."

 

[neptronix]

Here's something to think about.
ChatGPT is probably at least 8% better at thinking about batteries because of people on this forum.
ChatGPT is nothing without the 30 trillion or so words it's trained on, which happen to be provided on the public web, by us.

Like Soylent Green, AI is made out of people. Forums are one of the most scraped places on the internet because they contain some of the world's last genuine discussions. Reddit also makes up a high weight in USA models for similar reasons

I've had AI provide me lots of dumb and wrong answers for ebikes. I still think it's a better idea to ask the tribe.

A good KB can enable a few things with AI:
- can use a small ( very cheap ) AI model to perform searches faster, given the scope of text isn't 7gb, as it is on the forum, more like <100mb!
- because the dataset is small, the quality above average, using this as a sole source with a high weighting could boost accuracy in itself.
- one of these days, we will have the AI hardware to train and run a small model that can provide highly accurate results and blow away any commercial AI due to the very specialized nature.

Most importantly to me, if/when we build a better AI search engine, i want it to be able to attribute forum members for what's quoted. Right now, ChatGPT is taking people's work without crediting them. That's pretty discouraging to some people, that as a result of their work, technology replaces the credit.

I like deepseek because it's always given me citations to the webpage it's referencing. Love it because i can follow up and read more from the source, or double check the AI's reasoning process. A good forum search engine should do the same.

 

[E-HP]

I’m not asking it battery questions, but asking it to analyze how the topic has been supported by ES over time. With the rules in place, it provides examples to support its conclusions as well as rate its own response. Its analysis seems to pass the smell test with respect to the types of support.
I’m just saying for certain categories of problems, the “typical technical answer” can lead to catastrophic results, even if the answer is technically correct based on the information provided, when assuming the information provided is accurate and complete.

Approach	What it tends to look like
Amberwolf-style support	Establish the facts; challenge assumptions; identify possible failure modes; request measurements; interpret measurements; recommend action
Typical technical answer	Identify likely problem; explain likely cause; suggest component/setting/fix
Specialist answer	Go very deep into one particular technical issue
Product-oriented answer	Recommend a BMS/cell/charger based on the stated requirements

Dogman Dans E bike burns his house.

First of all, an apology for just vanishing like that. I told the moderator crew what happened last February, but I was hesitant to talk all about it until I had money in hand. And I had quite a lot to do, with a good dose of PTSD. The short version, my cheap ass shipped illegally 18650 48v...

endless-sphere.com