CGPT >> KB

[E-HP]

I've been playing with ChatGPT for a few months. It's interesting. For ebike related info, its hit or miss for providing the correct response in one iteration. I've found that it will provide different or additional information if you just tell it it's wrong. It will even refine it's answer if you just say you think it's wrong. You can tell if it's answer appear to be making certain assumptions, which appear to be incorrect, and it will dig deeper, or provide the assumptions, and if you tell it you believe the assumptions are wrong it will try to make corrections and provide a better answer. (It's also sensitive to name calling, lol; on another topic, I told it that it was racist, and it made a big deal about the ways the response might be construed as racist, but defending it and provide explanations or make adjustments.).

Anyway, on some topics I've had to call them on bad assumptions or errors almost up to a dozen times, but in the end the response if very well formatted, clear and logical. For me, this is nice, because you can give it questions, etc., without the grammar, spelling, punctuations, etc. do not need to be perfect, but the end product is clean and can be easily copied and pasted. This is way easier than typing it all out myself, and the process to get ChatGPT to provide a correct and well formatted answer isn't too hard.

I was thinking to try out populating one or two of the stubs in the KB with ChatGPT generated, but user reviewed, responses, but would want others to review to see if the response is acceptable, as an experiment, before going further. I think it's one way to get the bots to work in our favor, because I can tell in some cases where the information came from ES.

On one occasion, after pages of back and forth before a clean/correct response was given, I complimented it, and it humbly accepted the praise, but then complemented me for pointing out it's errors and bad assumptions along the way, lol.

 

[neptronix]

You've already understood the problem with AI:
If you don't know the topic, it sounds super smart.
If you know the topic, you find yourself correcting it a lot or nudging it in certain ways.

Most new AI users arrive on 'mount stupid' extremely quickly. This is an analogy for The Dunning-Kruger effect.



Once a user understands AI limitations and knows it's strengths and weaknesses, it does become useful.
A person skilled in the topic's job is now to be prompter, fact checker, and grammar editor.

I've found this to be true in systems analysis and programming. But it took a while.
With writing documentation, i find that 'cross breeding' with other articles as context for style, a little pile of information on the thing, and an outline... after a human final pass, the results are better than what i'd write.

If we allow people to use AI to write knowledgebase articles, it's important to know what part of the curve they are on.



I think if someone were to demonstrate they knew how to wield AI to generate high quality information, they could get a stamp of approval from me.

I would love to build some kind of context retrieval system that would synthesize data from ES for you and make your job even easier, but we don't have the AI hardware to do it yet.

Conditions:
- human inspired and fully human checked, to prevent slop
- not too wordy like AI output tends to be
- same quality as an average human written article
- it lacks robotic tone, unnecessary lists, emojis, nonsensicality, errors, wrong emphasis, and other signs of "nasty looking spatter on it's welds", which indicates a human either directed it well, and/or did the appropriate cleanup work.

 

[E-HP]

You've already understood the problem with AI:
If you don't know the topic, it sounds super smart.
If you know the topic, you find yourself correcting it a lot or nudging it in certain ways.

Most new AI users arrive on 'mount stupid' extremely quickly. This is an analogy for The Dunning-Kruger effect.

View attachment 389555

Once a user understands AI limitations and knows it's strengths and weaknesses, it does become useful.
A person skilled in the topic's job is now to be prompter, fact checker, and grammar editor.

I've found this to be true in systems analysis and programming. But it took a while.
With writing documentation, i find that 'cross breeding' with other articles as context for style, a little pile of information on the thing, and an outline... after a human final pass, the results are better than what i'd write.

If we allow people to use AI to write knowledgebase articles, it's important to know what part of the curve they are on.

View attachment 389556

I think if someone were to demonstrate they knew how to wield AI to generate high quality information, they could get a stamp of approval from me.

I would love to build some kind of context retrieval system that would synthesize data from ES for you and make your job even easier, but we don't have the AI hardware to do it yet.

Conditions:
- human inspired and fully human checked, to prevent slop
- not too wordy like AI output tends to be
- same quality as an average human written article
- it lacks robotic tone, unnecessary lists, emojis, nonsensicality, errors, wrong emphasis, and other signs of "nasty looking spatter on it's welds", which indicates a human either directed it well, and/or did the appropriate cleanup work.

People might be able to get past mount stupid, simply by asking are you sure? or that doesn’t seem right. I think that forces it to look at how much of the response was built on assumptions, and look further into them.

I think it’s a smart writing aid for now, and you can see from the response on ebike stuff, where you need to correct an assumption, and correcting the assumption corrects the response.

I decided to see how it works on a topic I have little interest typing up, ebike lighting. I asked “how to add lighting to an existing ebike. The initial answer wasn’t bad, and it provided adding battery/rechargeable lights, e-bikes with controllers with a light output, and DIY with adding a buck converter. For the latter, it provided a rudimentary diagram of the components and where they are in the circuit relative to the other components (battery, switches, buck, lights). Also mentioned lights that work at full ebike voltage with built in drivers as an option, removing the buck from the diagram. Then I asked about using a specific factory ebike (Lectric), where the controller was being upgraded, and how to utilize the existing lights. That was OK but needed more work, and it had a hard time with a functioning brake light switched via the brake cutoff switches. That will take more work to get it to write what I want out of it. After 4 back and forths, the response looked somewhat acceptable.

I’ll try filling out a stub, and mark it CGPT assisted until it’s fully vetted.

PS. I asked it about some stuff about finding a local restaurant that had disappeared (it was actually a pop up that does business in an existing bar or establishment). It took a while to get good info from it (checking licenses and county record, etc.) but the end response made sense. I asked the exact same question a couple of weeks later, using a different device (without a ChatGPT login), and the response was the cleaner response I’d gotten near the end of the other session, so it incorporated what it “learned” from the earlier session.

 

[neptronix]

People might be able to get past mount stupid, simply by asking are you sure? or that doesn’t seem right. I think that forces it to look at how much of the response was built on assumptions, and look further into them.

What if they don't know enough to ask that and it's still wrong?
This has happened to me, had many arguments with the clanker. Luckily i know what i'm doing, otherwise i'm shippin' slop

I decided to see how it works on a topic I have little interest typing up... ...That was OK but needed more work, and it had a hard time with a functioning brake light switched via the brake cutoff switches. That will take more work to get it to write what I want out of it. After 4 back and forths, the response looked somewhat acceptable.

Yep.
For good results you want to come up with the outline yourself and also set the tone.
For best results, you also include some example human-created documents as the tone reference. This is how i get AI to write documentation like me ( i just need to do a few documents by hand to my best effort first )

I’ll try filling out a stub, and mark it CGPT assisted until it’s fully vetted.

Cool
Right now there is a little bug where it's picking up the funding notice on the page, don't mind it, we'll fix that.


If you want to set a draft state, set the visibility to 'editor'

PS. I asked it about some stuff about finding a local restaurant that had disappeared (it was actually a pop up that does business in an existing bar or establishment). It took a while to get good info from it (checking licenses and county record, etc.) but the end response made sense. I asked the exact same question a couple of weeks later, using a different device (without a ChatGPT login), and the response was the cleaner response I’d gotten near the end of the other session, so it incorporated what it “learned” from the earlier session.

I can ask different models the same question and get different answers depending on how the question is worded.
That's because there is a randomization function ahead of the text generation.

See it a lot with programming where i'm asking something obscure but it sometimes knows what i'm talking about, other times not. Luck of the draw.

Some paid versions of online services do offer a 'memory' and what you are describing works fairly well from what i hear.

 

[lnanek]

People might be able to get past mount stupid, simply by asking are you sure? or that doesn’t seem right. I think that forces it to look at how much of the response was built on assumptions, and look further into them.

Unfortunately, most AI offerings are just large language models trained on public data to produce the next most likely text sequence given a prior one.

It's not that it goes and looks into assumptions when it is wrong. It just tries to predict the text of someone who did that. None of the training data that was used in the beginning changed. None of the model generation that locked down how the variables generated from the training data relate to each other changed. It's just auto-completing a longer prior text sequence than before.

People who don't know the answer already would likely throw it off as much as help it.

 

[E-HP]

Unfortunately, most AI offerings are just large language models trained on public data to produce the next most likely text sequence given a prior one.

It's not that it goes and looks into assumptions when it is wrong. It just tries to predict the text of someone who did that. None of the training data that was used in the beginning changed. None of the model generation that locked down how the variables generated from the training data relate to each other changed. It's just auto-completing a longer prior text sequence than before.

People who don't know the answer already would likely throw it off as much as help it.

I’m just adding it to my workflow as a more efficient way of typing. I can ask questions, comment, etc. in very few words, one finger typing on my tablet and eventually get the result I want to paste in. A lot less typing so I don’t have to drag out my laptop.
We’ll see if the results pass the smell test on review.