Robotics Project: Snuggles, The Wolf

[amberwolf]

what does slow mean in your case?


i don't know how fast the imus can internally process data to spit out processed stuff, but they spit out the current raw data in realtime. ther'es probably some microseconds delay, but not enough to worry about most likely. processing the data at the main system will almost certainly take much longer than generating it does.

some of the stuff i've read about the various imu boards/chips that have internal processing ability indicate up to 100ms to spit out processed data, which is too slow for realtime positioning feedback, but i would just use the raw datastream in those cases to the local mpus that are doing the motor control.

thats part of the reason for distributed processing with tiny mpus; each one can take care of just one thing so it doesn't have a big long program handling lots of tasks, just one small subset of tightly integrated tasks that depend on the same dataset / feedback / command process, like running just one motor or just one limb, etc.

if the main processor that has to make all the behavioral decisions also had to process all the individual data and motor control all in realtime, it would probably not be able to do it reliably or at all, or the program's interrupt structure might be very complex and hard to code and troubleshoot. but if all the little stuff is handled locally by separate mpus they don['t have to be super powerful to deal with just one subset of sensor data and one or two motors. they just get told to do a particular thing by the main mpu and then do that. overall the code might end up more complex as in more separate programs running, but each one should be simpler to create and troubleshoot problems in as it's designed. more on this in a separate post



either way, the imus are necssary--remember that in my proprioception network (post on the first page describing its basics) the imus do multiple things

their first and most important dataset is the accelerometer response to vibrations, impacts, etc as all the physical input is detected this way because other normal touch and force sensors can't work under the fur correctly or at all

their second dataset is the position sensing, and movement sensing; some of the first dataset gets used to determine rate of change of movement as well.

 

[Guerney]

Offload some processing to a high powered PC via ethernet? You could build an ethernet cable into the robot dog's tail? I don't want to imagine where the logical place for the ethernet port would be.

 

[amberwolf]

Offload some processing to a high powered PC via ethernet? You could build an ethernet cable into the robot dog's tail? I don't want to speculate where the logical place for ethernet port would be.

no reason to use the tail for that, the tail has to be a tail and do all the things a tail does, or else it's not dog or wolflike and doesn't do what's required.

as previously described in the thread all the motors, etc will be external and so will the processing except for those things like the imus that have to be phsyically embedded in teh wolfybot. so there will already be an umbilical (a rather large one), but it wont' be the tail. it will rpobably come out of the "butt" under the tail.

if there was any way to actually put all this stuff inside it along with a big enough power source, and have it all be quiet, and not melt or catch fire, that'd be great, but i haven't figured out any way to do that. remember that it's a wolf (or dog), and is covered in fur, and have foam or other padding under that to make it snuggly and squishable, so it will be very well thermally insulated, and it's highly likely to cause failure of all the things that generate heat, like motors and cpus and stuff. power source would pretty much have to plug in the wall to run all this stuff (guessing at least a thousand watts overall at full load, probably a lot less when laying still "asleep"), a battery that would fit inside along with all the stuff wouldn't be enough to run it for even an hour at a time.

even though the actual size of the wolfy is pretty huge (real life size), the only place to fit anything is the ribcage area, which is only a small volume, perhaps half that of a tidycats litter bucket, or perhaps 2 loaves of bread; the ribcage itself has to be flexible like a real one in order to move correctly so it's not nearly as much space inside . the skull space is already going to have the mechanics for moving hte ears and eyes and jaw and any other face parts i can manage to make work and still be squishable without breaking. the limbs have no space for anything as the pull cables for the motors to move them run thru them. if t he cables have to run outside the hollow tube bones then there will be space inside those if they're metal but ideally i'd like to print the bones to shape them similarly to real ones for the more realistic movement abilty and feel under the padding. the belly needs to be squishable and flexible like a belly, and cant' really have big solid pieces in it even if some stuff does go in there because the body has to be able to curl up and flex around realistically which it can't do if the belly has a big solid mass in it that gets in the way.

originally i was planning to run the main code on a spare old laptop or two or old desktop/towers i have hardware to build up, but now i've got some donated old hp rack servers that are much more powerful with way more ram that can be used for that. probably only take one to do everything, even if i have some form of ai for behavioral learning/response.


the umbilical will go to a box (probably rather large) that's soundproofed as well as possible given that it has to be cooled, that will hold all the motors and computing / processing. teh cooling will be ducted past/thru anything that needs it, to minimize the amount of fans that need to be used, and those fans will all be as large and slow as possible to give sufficient airflow with minimal noise. (done this before for recording-studio-type workstations, works great).

 

[Guerney]

Reduce the robot dog's internal heat and external noise by offloading some processing to a high powered soundproofed PC via WiFi? Some WiFi spec or other might be fast enough perhaps? I 've never hugged a dog, but I guess hugging a cold one could be unsatisfying, might even be concerning and unsettling. I don't want to imagine where the logical place the hot exhaust gases to exit the robot dog might be, not to mention how it could sound.

 

[amberwolf]

Reduce the robot dog's internal heat and external noise by offloading some processing to a high powered soundproofed PC via WiFi? Some Wifi spec might be fast enough perhaps.

the noise is mostly the motors, as already stated those and all the processing would already be in an external box.



all the servomotors have a lot of gearing so they can be small, so the motors run really fast (in short bursts), and the gearing and motor make quite a lot of whining sounds. if you soundproof those, the motor and/or it's electronics get hot and fail over time; if the gearing is plastic it gets soft and strips. so if soundproofed some other form of cooling is needed (there are a number of possible ways but they all increase complexity and risk of failure grows dramatically--some of the failure modes of some of the systems can cause catastrophic damage to the wolfy itself requiring disassembly and replacement of padding/covering).

the big spine/hip/shoulder/neck motors are likely to be steppers instead of servos, as these can be easier to use to hold a position with less stress on the motor/driver, especially if using a gearbox (which i might have to do to keep the motor and driver size down). these also get hot, and are not all that quiet either (and any gearbox makes them louder).

i don't want to hear robotic sounds inside the wolfy, so....external soundproofed box for everything that gets hot or makes noise.



wifi might be fast enough, but there's not likley a need for it, the majority of stuff already interconnects via i2s or spi or other serial buses so has to be wired up via the umbilical. unless all that data is internally wired to a central mpu that then concatenates it into a stream that can go via some other connection/protocol to the external processing.

somewhere else in the thread there should be some mention of the possiblitiy of using hte wifi that's on many of the esp32 mpus i have to do the interconnect to the external processing.

the problem with having a lot of layers of comm protocols is that you can get comm failures or data loss at any of htem, so the more layers there are the harder it is to troulbeshoot and fix.

I don't want to imagine where the logical place the hot exhaust gases to exit the robot dog might be, or how it could sound.

except for breathing or panting, moving air in and out the mouth and nose, whcih there isn't a practical way to actually do with any really significant air movement, there's nowhere to vent from, or circuilate air thru. i worked out a bunch of possible ventilation systems about 30 years ago when i wanted to build a realistic-wolf-shaped pc case that would have been covered in fake fur and have a few wolf behaviors, though it would only have just sat upright and not move around except for a head that would pivot around to "watch" things and a jaw that would open and close with a tongue that could move in/out to simulate panting when the computer was hot. most of the systems would have had to pull air thru the fur on the sides of the case and push it out the open mouth. some of them would have had belly slits to do it, one had a raised base that the wolf sat on that would have the vent slits in it. i never built the case partly because cooling was never sufficient in the realistic-looking ways i preferred, and partly because it was too expensive for the robotics parts needed to do the head movements and stuff. (i hadn't yet learned of all the alternate parts sources i know about today, salvaging stuff for free out of old printers, etc).


unfortuantely any notes i had locally stored about that project were lost in the housefire and/or the subsequent cleanup.. i had posted about it along with pics and sketches on an old version of the themanime forums in some discussion about interesting computer cases, but that version was hacked and destroyed by spambots or whatever, and completely lost as they didn't have any backups, and the wayback machine doesn't have copies either. so all i have left of that is whatever memories haven't fallen out of my head.

 

[Guerney]

i don't want to hear robotic sounds inside the wolfy, so....external soundproofed box for everything that gets hot or makes noise.

How precise does movement have to be for a prone huggable wolfy? Use hydraulics for movement via cogs and gears instead of whiny servos, powered by external soundproofed motors? Cog and gear noises should be easier to dampen. If the resulting wolfy is too cold to hug, use hot hydraulic oil? The pipes could go up through the bed and mattress.

 

[amberwolf]

you'll have to trust me that i have already thought out the stuff to do this with.

i don't need help redesigning it.

i need help implementing the design ideas i already have.


hydraulics (anything using liquids) leak.


if they leak into the padding and body i have ot throw the whole thing away and start over, and it is a massive amount of work to do all that, and to make it look the wya i want it to.

plus tthat would be traumatizing, as if it had died.


heat for realism is mostly not necessary--fur (fake or real) reflects body heat well enough that unlike plastics and whatnot it doesn't feel cold to the touch in normal room temperatures.

for any parts that might need heat, like cast silicone paw pads, noses, maybe ears/face, maybe belly, it's very simple to add flexible wire heaters into those in short paralleled segments with thermal regulation controls (so that if one segment fails the rest still work). already done that on the non-robotic versions experimentally over the years.

 

[amberwolf]

fwiw, while i appreciate the alternative points of view, and the thought that went into them, if i have to explain every alternative that's already explored and why it can't be used, this thread is going to be a few thousand pages long, and take several years to write (or longer, possibly as long as it has taken me to come up with, research, ponder, and design them all). and all that time wasted instead of being spent on actually getting a working system.


so i am writing up the things i already "know" will (should) work, and the requirements for what the system and/or end-unit has to do or be able to do.


so for those that want to help, the best thing they can do is to help me implement the systems already devised.

 

[amberwolf]

thoughts on an example limb processor; first the limb has to be defined


define x / y / z axes overall as

x = side to side, or yaw

y = up and down or pitch

z = front to back, or forward / reverse


so any motor that moves things along the length of the wolfy is z axis, if it moves it to the side it's x axis, and if it moves it up or down it's y axis.


there are five motors per limb.

at the shoulder / hip there is both a z and x motor that rotate a heim joint (in place of a full ball joint to give motion limits without the skeletal muscles and tendons that normally restrict it.

the elbow only has a z motor; it only pivots in one way.

the "wrist"/ankle also has a z and x motor, but the x motor doesn't exaclty move those side to side, isntead it twists the radius/ulna / tibia/fibula at the paw end to change the angle at which the z motor does it's work, similar to the way that the skeletal muscles would.

this allows it to move the paws (especially the front ones) for all the things like gripping / grabbing / hugging they do, and the motions necessary to balance when sitting up, standing, getting up from a side-laying position, etc.

also for walking if i ever manage to do that (doubtful, not a requirement for this wolfybot since it's just meant to stay right there and be with me (or other users that choose to build one or have one built for them) and be cute and otherwise emotionally supportive and destressing).


in an ideal world each toe would also have a z motor, but i don't think i can fit that many cables thru the limb paths, and that would at double the umbilical size too. so most likely the toe mechanics would simply be built to move with the wrist/ankle z motor movements but with a slacking/spring mechanism that allows the toes to spread under load when pressed against a surface even when the wrist/ankle motor is commanding contraction, or curl up if not against a surface. this is likely to take a number of iterations of design and build to get to work as desired, but the first simple implementation will probably just be looser joints in the toes (ovoid holes for the pins instead of round) and elastic bands along the ventral surfaces of the toe bones (or just the silicone of the toes molded in the contracted position).



there would be an imu for positioning / feedback attached to each of the skeletal segments of the limb, so one on the shoulder-plate / pelvis, one on the humerus/femur at the elbow end, one on the tibia/fibula / radius/ulna pair (shouldnt need one on each bone) at the paw end, and one on the ventral surface of the paw "bones" under the main "pad".

there will be additional imus as needed for vibration/touch detection, if that avialable from the ones above are insufficient for sensitivity required.


a single mpu can probably handle the input from all the positioning imus and process it along with the motor command processing, but separating those functions can be done if needed. it probably isn't necessary to subdivide it any further per limb.

the limb mpu would get a command from the main behavioral mpu (mbm) that tells it to extend forward, retract backward, stay, move laterally, etc., to place the paw in a specific spatial location, and the limb mpu would use the imu data it has, plus the full-body-coordinates the main sensory mpu (msm) keeps track of (from all the local data vs a power-up calibration set), and do the math to move each motor the right way to get the paw where it should be.

it would also have a movement model to base those on, so that all movements are realistic. there would be various modes, such as hyper, sleepy, playful, bashful, etc., that it would refer to once the mbm has told it which mode ot use, so that the limb mpu moves the motors quickly, slowly, etc., to match the behavior at the time.

 

[mxlemming]

Hi, you pointed me to this thread... 5 pages I have skimmed but probably missed a lot.

If I was to embark on this, I would probably be following a similar architecture - ESP camera, control board and link to higher level processing board. However, I would probably be pushing for something better than an arduino uno... and would be avoiding the ARduino environment. It starts out good with the demo examples, but later gets irritating and restrcitive and has no debug ability. "Lob code over the wall and see what happens, then guess the errors".

I would definitely go towards a realtime processor and proper IDE. I would choose something for which there is a huge support network and which is quite readily portable between devices... This basically means STM32.

I generally default to the stm32f405, which happens to be the VESC MCU... because it is the cheap very well understood and yet has all features model.

With that in mind, I asked Chat GPT...
"If I have an stm32f405rg with access to all the pins, and want to connect 4 RC servos (1000-2000us 50Hz ppm signal) two step direction steppers and 4 MPU6050 accellerometers over i2c, could you suggest appropriate pin connections?"
Then
"Can you redo this assuming that I would like to use USB on pins PA11 and 12?"
Then
"I think for 4 MPU6050, we will require an extra I2C line since the AD0 pin only allows for two addresses if I understand right... Is there one available?"
It spat out after the prompts:

Summary of Pin Connections (Final Updated):​

Component	Pin	STM32F405RG Pin	Function
Servo 1 PWM	PWM	PA0	PWM (TIM2_CH1)
Servo 2 PWM	PWM	PA1	PWM (TIM2_CH2)
Servo 3 PWM	PWM	PA2	PWM (TIM2_CH3)
Servo 4 PWM	PWM	PA3	PWM (TIM2_CH4)
Stepper 1 Step	Step	PB0	GPIO
Stepper 1 Dir	Direction	PB1	GPIO
Stepper 2 Step	Step	PB2	GPIO
Stepper 2 Dir	Direction	PB3	GPIO
MPU6050 1 SDA	Data	PB7	I2C1
MPU6050 1 SCL	Clock	PB6	I2C1
MPU6050 2 SDA	Data	PB7	I2C1
MPU6050 2 SCL	Clock	PB6	I2C1
MPU6050 3 SDA	Data	PB11	I2C2
MPU6050 3 SCL	Clock	PB10	I2C2
MPU6050 4 SDA	Data	PB11	I2C2
MPU6050 4 SCL	Clock	PB10	I2C2

Great, we know where to connect to physically. Can we enter this into the STM32 Cube environment? Well, after some piffling about with the ST code repo finder in help stm32cube updates and setting it up with the ST account and (ugh this used to be easier)... we get to the IOC...


Set the clock to 8M (usually for a board depending what you buy) and set the HCLK (main frequency) to 168. Press enter and it does it.

So let's set up the timer2 for th RC servos
Ask ChatGPT for some ARR and PSC values... and set pwm generation.

After telling CGPT we use 168M HCLK, and nudging it:
"For the tim2, what ARR and PSC should I use?"
"Can you redo that to have microsecond resolution?"
"Can we redo that assuming the HCLK is 168MHZ?"
we get:

Summary of Timer Settings:​

Parameter	Value
PSC	167
ARR	19999
Timer Frequency	1 MHz (calculated from HCLK divided by PSC)
PWM Frequency	50 Hz (calculated from ARR)
Resolution	1 µs per tick (due to 1 MHz timer frequency)

So we enter that...



We do similar for the I2C:


And we generate the project... which we see compiles.


So we now have a building stm32 project that has 4 available PWMs and 2 I2C channels. How do we use them? Well... we ask ChatGPT... it will happily make you simple drivers to read MPU and set the servos to twiddle.

I shall not go further with this. I do not have hardware. Good luck This is how I would approach it. Then similar for the ESP camera and the higher level stuff.

You can add USB/USART/...

 

[amberwolf]

Thank you for all that; I hadn't tried breaking it down that way to the AIs I have tried to "talk to" about this...so the results I got back were as garbage as the input I gave them.

IIRC the RPI stuff is STM32?

(I think some of the arduino may also be STM32, but the coding is different?)

Just thinking out loud...because I kind of "need" to use as much prebuilt hardware as I can. I don't have the budget*** to have things built for me custom (unless that sort of thing is an awful lot cheaper than it used to be), so I am pretty much stuck with having to use existing cheap MPU modules that have as much of the I/O and other hardware on them as I need, as much as possible.

I also don't have much ability anymore to build tiny things with SMT parts (it's hard enough to handle big thru hole parts, even with my camera-magnifier workstation), so anything I don't have to build myself makes it that much more likely I can do various things.

If it ends up requiring a completely custom-designed and built hardware set to do this job, then that will have to be made to happen, but my initial development is realistically going to have to use existing cheap modules.


But I can still try to use the process you've shown me to create the system from the existing modules, and the code to make them do their various jobs.





***I have a bunch of various technologies (ebike, music, computer, etc) around here that I don't use that could theoretically be sold to generate funds, but AFAICT almost none of it is actually worth $$ to anyone (whch is why I ended up with it in most cases), so I have not tried (in a while) to sell it off (I couldn't even give away any of the stuff I had to clear out the last few years).

 

[AnthonyC]

As an aside Thanks mxlemming you pretty much answered a prayer!

Mature STM32F405RG development board for trustworthiness and support
Similar cheaper STM32F405RGT6 board for subsequent needs
Intro to STM32 Cube environment
Opinions on STM32 Cube environment. Suits pros and keen amateurs AIUI.
MicroPython for STM32. Easy prototyping, also specific uses e.g. NASA. Runs on Pi Pico etc.

RPI uses Broadcom or their own microcontrollers, but like many micros they're supported by many languages, and many IDEs host multiple languages too.

 

[mxlemming]

Thank you for all that; I hadn't tried breaking it down that way to the AIs I have tried to "talk to" about this...so the results I got back were as garbage as the input I gave them.

IIRC the RPI stuff is STM32?

(I think some of the arduino may also be STM32, but the coding is different?)

Just thinking out loud...because I kind of "need" to use as much prebuilt hardware as I can. I don't have the budget*** to have things built for me custom (unless that sort of thing is an awful lot cheaper than it used to be), so I am pretty much stuck with having to use existing cheap MPU modules that have as much of the I/O and other hardware on them as I need, as much as possible.

I also don't have much ability anymore to build tiny things with SMT parts (it's hard enough to handle big thru hole parts, even with my camera-magnifier workstation), so anything I don't have to build myself makes it that much more likely I can do various things.

If it ends up requiring a completely custom-designed and built hardware set to do this job, then that will have to be made to happen, but my initial development is realistically going to have to use existing cheap modules.


But I can still try to use the process you've shown me to create the system from the existing modules, and the code to make them do their various jobs.





***I have a bunch of various technologies (ebike, music, computer, etc) around here that I don't use that could theoretically be sold to generate funds, but AFAICT almost none of it is actually worth $$ to anyone (whch is why I ended up with it in most cases), so I have not tried (in a while) to sell it off (I couldn't even give away any of the stuff I had to clear out the last few years).

RPi is some other higher end MCU that is hideously complex to use at this level, so we end up using them with an operating system on top. There is the Rpi pico, but it is really quite low power. It's nice that they made their own MCU, but as far as I can see, it is far worse than the f405RG I suggested.

The boards AnthonyC found look reasonable for the stm32. Of course you can follow the ChatGPT methods with any boards. Micropython store one has decided to hide the SWD pins (they are PA13 and PA14) for debug btw.

Wiring the accellerometers and servos to this board with twisted sets of wires is fine. Have to be careful with any MCU not to go shorting 5V/power to io and grounds etc. MCUs are fragile. Sure you are familiar with this, I recall you have decapitated a few controllers to lebowski them so you're probably fine on that front.

 

[amberwolf]

Thank you both very much.

I've done electronics stuff for a long time (mid 1980s?), so plenty of experience with blowing things up (and knowing what not to do, to prevent that :lol. Learned a lot about ESD prevention when I worked at Honeywell CFSG in final test/QC on the A320 and MD80/88 lines. (we even had special floors). I actually have a partial roll of RG58 and another of RG6? shielded coax that can be used for critical signals if necessary.

I expect that physical actual noise and vibration, corrupting the IMU data as it's acquired, is going to be more of a problem than electrical noise.

My mistake about STM vs RPI; I hadn't fully read the results of my quick searches about MPUs after reading your post. Looking at them now, despite what some of google's search results show, the various pages are asking which one to go with--RPI or STM32.

I will take a look around for what cheap STM32 boards exist that already have as much of the other interconnectivity/etc on them as I can get, with the MCU version youve pointed out. Maybe a compromise can be made if the STM32 code can be run better / faster than the RPI or Arduino code would over their OS, so that fewer units will be needed to process the IMU data, and I can spend more on them. I can still use the handful of assorted ESP32 (one with a cam), RPI Pico, and Arduino nanos I already have for other things, or for parts of this system that are suited for them.

In the meantime, in between involuntary naps today, I tried out ChatGPT with some not-well-thought-out questions and even so still seem to have gotten MUCH better results than the last time I tried that. (IIRC, it refused to provide any kind of detailed results, no code, etc). So they've changed and improved it significantly since then (2022? 23?); it even understood what I was asking it for, AFAICT, better than most of the humans that I've discussed this with. :lol:

Subsequent posts will have the chat and code resulting from that, to archive them here in case they end up being useful later on.



Oddly, more than a decade ago (I think), I attended a renaisys (sp?) demo where they gave out a couple of STM8(?) "usb key" sized units (that I still have), along with the STM32 Circle (which I think I still have, but isn't with those for some reason). At the time I had planned to build a complete ebike control system that would've been a bit like some of these highly-integrated OEM ebikes that have come out the last few years (but didn't really exist then, other than maybe BionX). It would have been open source so anyone could build one from it. There might even be a thread here for it, but it was probably before I came here. I couldn't do the coding and tried to find help doing that, but had no luck then, and eventually I had given up on it because I didn't really need it, was just a nifty idea, and I had way too much else to do (including some of the early research into MPUs and IMUs for this wolfy project, which was much more important and complicated). Somewhere in there the housefire happened and I mostly forgot about the bike MCU system and these STM units I had.....

But if the STMs that are on the keys and/or Circle (if I can find it) are good enough, maybe I can use them to test-run some of the code bits with the IMU and other test hardware I have now.

 

[amberwolf]

The following posts are the raw output from the ChatGPT session; I haven't done any verification of any of it, other than to see that the diagram it provides as an image is probably not useful. The quantitites I gave for MPU6050s and RPIs was arbitrary.

When I get a chance and am not so tired I'm likely to make mistakes, I'll hook up some parts and try out the code snippets provided. (I won't know whether any of them are working other than to see their results or lack thereof, until I learn how the code itself actually works).

connection diagram for mpu6050s to rpi picos v1

show me the wiring diagram to connect 36 MPU6050 boards to a set of RaspberryPi Nano boards to process the IMU data
ChatGPT said:
ChatGPT
To connect 36 MPU6050 IMU boards to a set of Raspberry Pi Pico (not Nano) boards and process their data, you'll need a scalable architecture, as the I²C bus used by the MPU6050 has address limitations. Below is a high-level overview and guidance for such a setup:

Requirements
Hardware:
36 MPU6050 boards
Raspberry Pi Pico boards (quantity depends on configuration)
Pull-up resistors (for I²C lines)
Power supply
Connecting wires
Tools:
Soldering kit (if required)
Multimeter (for troubleshooting)
Challenges & Considerations
I²C Address Limitation:

Each MPU6050 has two possible I²C addresses (0x68 and 0x69), selectable via the AD0 pin.
Each I²C bus can handle a maximum of 2 MPU6050 boards.
Power Supply:

Ensure the power supply can handle the combined current draw of all MPU6050 boards and Raspberry Pi Pico boards.
Scalability:

Use multiple Raspberry Pi Pico boards to divide the workload.
Proposed Architecture
Divide the MPU6050s Across Multiple Raspberry Pi Pico Boards:

Each Raspberry Pi Pico manages 6 MPU6050 boards.
For 36 MPU6050 boards, you'll need 6 Raspberry Pi Pico boards.
Data Collection and Processing:

Each Raspberry Pi Pico reads data from its assigned MPU6050 boards.
The Raspberry Pi Picos send processed or raw data to a central system via UART, SPI, or another communication protocol.
Wiring Diagram
Per Raspberry Pi Pico (6 MPU6050 Boards)
Power:

Connect VCC of all MPU6050s to a 3.3V or 5V power supply.
Connect GND of all MPU6050s to the Raspberry Pi Pico GND.
I²C Connections:

Connect all SCL lines of the MPU6050s to the Raspberry Pi Pico GP5 (or another I²C clock pin).
Connect all SDA lines of the MPU6050s to the Raspberry Pi Pico GP4 (or another I²C data pin).
Use pull-up resistors (typically 4.7kΩ) for SCL and SDA.
Address Configuration:

Set AD0 to GND for 0x68.

Set AD0 to VCC for 0x69.

Assign 2 MPU6050s per I²C bus and configure each with a unique address.

Communication with Central System:

Use UART, SPI, or USB to communicate with the main processing unit.
Overall Connection
Power Distribution:

Use a central power supply for all Raspberry Pi Picos and MPU6050 boards, ensuring proper current distribution.
Data Flow:

Raspberry Pi Pico boards send IMU data to a master controller (e.g., Raspberry Pi or PC).
Diagram Explanation
While I can describe this setup in detail, creating a visual wiring diagram would be more illustrative. Would you like me to generate one for this configuration?






You said:
Yes, please.
ChatGPT said:
ChatGPT


Here is the wiring diagram showing the connections of 36 MPU6050 boards to 6 Raspberry Pi Pico boards, with all the required details and labeled connections. Let me know if you need further clarifications or adjustments!

 

[amberwolf]

rpi code to acquire mpu6050 data v1

Please provide the code for the MPU6050s onboard MCUs to filter their data. I need one set of data that is just the realtime XYZ axis position data, and a second set of data that is just the change in state from the previous readings, and a third set of data that is just the accelerometer 3axis data.
ChatGPT said:
ChatGPT
Below is the Python code for the Raspberry Pi Pico (or any microcontroller running MicroPython) to interact with MPU6050 IMU sensors. This program will filter and provide the three requested data sets:

Real-time XYZ axis position data: Raw gyroscope data.
Change in state from previous readings: Differences between consecutive gyroscope readings.
Accelerometer 3-axis data: Raw accelerometer data.
Python Code for Raspberry Pi Pico
python
Copy code

from machine import Pin, I2C
import utime

# Constants for MPU6050
MPU6050_ADDR = 0x68
ACCEL_XOUT_H = 0x3B
GYRO_XOUT_H = 0x43
PWR_MGMT_1 = 0x6B

# Initialize I2C bus
i2c = I2C(0, scl=Pin(5), sda=Pin(4))  # Adjust pins as per your wiring

# MPU6050 Initialization
def mpu6050_init():
    # Wake up MPU6050 from sleep mode
    i2c.writeto_mem(MPU6050_ADDR, PWR_MGMT_1, bytearray([0]))

# Read raw data from MPU6050
def read_raw_data(register):
    high = i2c.readfrom_mem(MPU6050_ADDR, register, 1)[0]
    low = i2c.readfrom_mem(MPU6050_ADDR, register + 1, 1)[0]
    value = (high << 8) | low
    if value > 32768:
        value -= 65536
    return value

# Get accelerometer and gyroscope data
def get_mpu6050_data():
    accel_x = read_raw_data(ACCEL_XOUT_H) / 16384.0  # Scale factor
    accel_y = read_raw_data(ACCEL_XOUT_H + 2) / 16384.0
    accel_z = read_raw_data(ACCEL_XOUT_H + 4) / 16384.0
    
    gyro_x = read_raw_data(GYRO_XOUT_H) / 131.0  # Scale factor
    gyro_y = read_raw_data(GYRO_XOUT_H + 2) / 131.0
    gyro_z = read_raw_data(GYRO_XOUT_H + 4) / 131.0
    
    return (accel_x, accel_y, accel_z, gyro_x, gyro_y, gyro_z)

# Main program
mpu6050_init()

previous_gyro_data = (0, 0, 0)
while True:
    accel_x, accel_y, accel_z, gyro_x, gyro_y, gyro_z = get_mpu6050_data()
    
    # 1. Real-time XYZ axis position data (gyroscope)
    realtime_gyro_data = (gyro_x, gyro_y, gyro_z)
    
    # 2. Change in state from previous readings (gyroscope delta)
    gyro_delta = (
        gyro_x - previous_gyro_data[0],
        gyro_y - previous_gyro_data[1],
        gyro_z - previous_gyro_data[2],
    )
    previous_gyro_data = realtime_gyro_data  # Update for the next cycle
    
    # 3. Accelerometer 3-axis data
    accelerometer_data = (accel_x, accel_y, accel_z)
    
    # Print all three sets of data
    print(f"Realtime Gyro Data: {realtime_gyro_data}")
    print(f"Gyro Delta: {gyro_delta}")
    print(f"Accelerometer Data: {accelerometer_data}")
    
    utime.sleep(0.1)  # Delay for readability

Explanation
Initialization:

The mpu6050_init() function sets up the MPU6050 to wake up from sleep mode.
Data Reading:

The read_raw_data() function reads 16-bit values from the MPU6050's registers and handles the 2's complement format.
The get_mpu6050_data() function retrieves accelerometer and gyroscope data, scaled appropriately.
Data Processing:

Real-time gyroscope data is output as is.
The difference in state is calculated by comparing the current gyroscope data with the previous reading.
Raw accelerometer data is scaled and output.
Adjustments:

Modify the I²C pins (GP4 and GP5) if needed to match your hardware setup.
Tweak the utime.sleep(0.1) interval based on your application requirements.
Deployment
Copy the script to each Raspberry Pi Pico.
Assign unique MPU6050 addresses to avoid conflicts.
Use serial output or another communication protocol to send data to a central controller.
Let me know if you need additional functionality, such as applying filters like Kalman or complementary filtering for smoother data!

 

[amberwolf]

rpi code to process mpu6050 data v1.txt

Thank you. Please provide the code to use the data from the above to work in the fashion of a seismic detection triangulation network, to take the data from all of the IMU sensors, one positioned on each bone of a wolf-shaped-robot skeleton, so that even the lightest touches on the fake fur covering the sensors/skeleton will be detected, and triangulate where each touch is and store that data in a table along with the system-time it was detected. Along with that, another set of code is needed to record the position and motion and speed of every part of the skeleton in relation to every other part. Code will be needed to filter out the noise from the motors that move the joints. The end result required is to model where and when each touch is, how hard each touch is, and also the position of every part of the wolf skeleton relative to every other part and it's motion speed and direction and rate of change.
ChatGPT said:
ChatGPT
This is a highly sophisticated project combining principles of robotics, kinematics, and real-time signal processing. Below is a modular breakdown of the system and corresponding Python code snippets.

System Overview
Hardware Setup:

IMU Sensors: Detect touch-induced vibrations and movements on each bone/joint.
Motor Control: Ensure motor signals are synchronized for noise filtering.
Central Processing Unit: Raspberry Pi or equivalent to collect and process data.
Tasks:

Seismic Detection and Triangulation: Determine the location and intensity of touches.
Kinematic Model: Compute position, motion, speed, and direction of all skeleton parts.
Noise Filtering: Remove motor-induced noise using techniques like adaptive filtering.
Data Storage: Log triangulated touches and kinematic data with timestamps.
Data Flow:

IMU sensors send real-time data to Raspberry Pi Picos.
Data is processed locally and sent to a central unit for triangulation and kinematic modeling.
Seismic Detection and Triangulation Code
Triangulation Algorithm
Use the Time of Arrival (ToA) of vibrations at multiple IMU sensors.
Calculate the location of the touch by solving for the intersection of spheres defined by the vibration propagation.
python
Copy code

import numpy as np
import time

# Define positions of IMU sensors on the skeleton
IMU_POSITIONS = [
    (0, 0, 0),  # Example: position of IMU on the skull
    (10, 0, 0), # Example: position of IMU on the front right leg
    # Add positions for all IMUs
]

# Speed of wave propagation through the skeleton material (m/s)
WAVE_SPEED = 500.0

# Record the timestamped accelerometer data from all IMUs
def detect_touch(accel_data, threshold=0.5):
    timestamps = []
    for imu_data in accel_data:
        if max(abs(imu_data[0]), abs(imu_data[1]), abs(imu_data[2])) > threshold:
            timestamps.append(time.time())
        else:
            timestamps.append(None)
    return timestamps

# Perform triangulation
def triangulate_touch(timestamps):
    valid_sensors = [(pos, ts) for pos, ts in zip(IMU_POSITIONS, timestamps) if ts]
    if len(valid_sensors) < 3:
        return None  # Insufficient data for triangulation

    reference_pos, reference_time = valid_sensors[0]
    intersections = []
    for pos, ts in valid_sensors[1:]:
        time_diff = ts - reference_time
        distance = WAVE_SPEED * time_diff
        sphere = {
            "center": pos,
            "radius": distance
        }
        intersections.append(sphere)
 
    # Simplified triangulation: average intersection points
    x, y, z = zip(*[sphere["center"] for sphere in intersections])
    return np.mean(x), np.mean(y), np.mean(z)

# Example usage
accel_data = [
    (0.1, 0.2, 0.3),  # Acceleration data from IMU 1
    (0.0, 0.0, 1.0),  # Acceleration data from IMU 2
    # Add data from all IMUs
]
timestamps = detect_touch(accel_data)
touch_location = triangulate_touch(timestamps)
print(f"Touch Location: {touch_location}")

Kinematic Model Code
Skeleton Kinematic Representation
Each bone/joint is treated as a segment in a tree structure.
IMU readings provide orientation (Euler angles or quaternions).
Combine orientations to compute relative positions.
python
Copy code

import math

# Define the skeleton structure
SKELETON = {
    "head": {"parent": None, "position": (0, 0, 0)},
    "neck": {"parent": "head", "length": 5},
    "front_left_leg": {"parent": "neck", "length": 15},
    # Add all bones and joints
}

# Update positions based on IMU data
def update_kinematic_model(imu_data):
    positions = {}
    for joint, data in imu_data.items():
        parent = SKELETON[joint]["parent"]
        if parent:
            parent_pos = positions[parent]
            length = SKELETON[joint]["length"]
            angle_x, angle_y, angle_z = data["orientation"]
            dx = length * math.cos(angle_x)
            dy = length * math.sin(angle_y)
            dz = length * math.sin(angle_z)
            positions[joint] = (parent_pos[0] + dx, parent_pos[1] + dy, parent_pos[2] + dz)
        else:
            positions[joint] = SKELETON[joint]["position"]
    return positions

# Example IMU data
imu_data = {
    "head": {"orientation": (0, 0, 0)},
    "neck": {"orientation": (0.1, 0.1, 0)},
    "front_left_leg": {"orientation": (0.2, 0.2, 0)},
    # Add orientations for all joints
}
kinematic_positions = update_kinematic_model(imu_data)
print(f"Kinematic Positions: {kinematic_positions}")

Motor Noise Filtering
Adaptive Filtering
Implement a notch filter or adaptive filter to remove motor-induced frequencies.
python
Copy code

from scipy.signal import butter, lfilter

# Butterworth Filter
def butter_lowpass_filter(data, cutoff, fs, order=5):
    nyquist = 0.5 * fs
    normal_cutoff = cutoff / nyquist
    b, a = butter(order, normal_cutoff, btype='low', analog=False)
    return lfilter(b, a, data)

# Example motor noise filtering
filtered_data = butter_lowpass_filter(raw_accel_data, cutoff=10, fs=100)
Data Storage
python
Copy code
import csv
from datetime import datetime

# Log data
def log_touch_data(touch_location, intensity):
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    with open("touch_data.csv", "a") as file:
        writer = csv.writer(file)
        writer.writerow([timestamp, touch_location, intensity])

def log_kinematic_data(kinematic_positions):
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    with open("kinematic_data.csv", "a") as file:
        writer = csv.writer(file)
        for joint, pos in kinematic_positions.items():
            writer.writerow([timestamp, joint, pos])

Next Steps
Integrate sensor data acquisition with filtering and processing.
Calibrate IMU sensors to reduce drift and error.
Optimize triangulation using advanced algorithms (e.g., multilateration).
Let me know how you'd like to refine or expand this setup!

 

[amberwolf]

rpi code to refine mpu 6050 data v1

Yes, I will need the code for those "next steps" too:
ChatGPT said:
ChatGPT
To refine the system and implement advanced capabilities for your robotic wolf skeleton, we can focus on these next steps:

Sensor Data Acquisition Integration
IMU Sensor Calibration
Advanced Touch Triangulation
Kinematic Modeling Optimization
Data Filtering with Kalman/Complementary Filter
Motor Noise Mitigation
Here's the extended code for these tasks:

1. Sensor Data Acquisition Integration
Efficiently gather data from all IMU sensors and centralize it for processing.

python
Copy code

import time

# Example to collect IMU data from multiple sensors
def collect_all_sensor_data(i2c_addresses, i2c_bus):
    sensor_data = {}
    for addr in i2c_addresses:
        try:
            accel_data = read_accel_data(i2c_bus, addr)
            gyro_data = read_gyro_data(i2c_bus, addr)
            sensor_data[addr] = {
                "accelerometer": accel_data,
                "gyroscope": gyro_data,
            }
        except Exception as e:
            print(f"Error reading from sensor {addr}: {e}")
    return sensor_data

# Replace with your IMU I2C reading functions
def read_accel_data(bus, addr):
    return (0, 0, 0)  # Mock accelerometer data

def read_gyro_data(bus, addr):
    return (0, 0, 0)  # Mock gyroscope data

# Example usage
i2c_addresses = [0x68, 0x69]  # Add all connected IMU addresses
while True:
    all_data = collect_all_sensor_data(i2c_addresses, i2c)
    print(all_data)
    time.sleep(0.1)

2. IMU Sensor Calibration
Gyroscope Offset Calibration
Eliminate gyroscope drift by calculating static offsets.

python
Copy code

def calibrate_gyroscope(i2c_addresses, samples=100):
    offsets = {addr: [0, 0, 0] for addr in i2c_addresses}
    for addr in i2c_addresses:
        for _ in range(samples):
            gx, gy, gz = read_gyro_data(i2c, addr)
            offsets[addr][0] += gx / samples
            offsets[addr][1] += gy / samples
            offsets[addr][2] += gz / samples
    return offsets

gyro_offsets = calibrate_gyroscope(i2c_addresses)

# Apply offsets during data collection
def apply_offsets(raw_gyro_data, offsets):
    return tuple(raw - offset for raw, offset in zip(raw_gyro_data, offsets))

3. Advanced Touch Triangulation
Multilateration Using Optimization
Solve the intersection of spheres more accurately using optimization libraries.

python
Copy code

from scipy.optimize import minimize

def multilaterate_touch(sensor_positions, distances):
    def objective_function(point):
        return sum((np.linalg.norm(np.array(point) - np.array(pos)) - dist) ** 2
                   for pos, dist in zip(sensor_positions, distances))
    
    initial_guess = np.mean(sensor_positions, axis=0)
    result = minimize(objective_function, initial_guess, method='L-BFGS-B')
    return result.x if result.success else None

4. Kinematic Modeling Optimization
Use Forward Kinematics to Model Skeleton
Refine positions of joints using joint rotations and link lengths.

python
Copy code

def forward_kinematics(joint_angles, skeleton_structure):
    positions = {}
    for joint, info in skeleton_structure.items():
        parent = info["parent"]
        if parent:
            parent_pos = positions[parent]
            length = info["length"]
            angle_x, angle_y, angle_z = joint_angles[joint]
            dx = length * math.cos(angle_x)
            dy = length * math.sin(angle_y)
            dz = length * math.sin(angle_z)
            positions[joint] = (
                parent_pos[0] + dx,
                parent_pos[1] + dy,
                parent_pos[2] + dz,
            )
        else:
            positions[joint] = info["position"]
    return positions

5. Data Filtering with Kalman/Complementary Filters
Kalman Filter Implementation
Refine accelerometer and gyroscope readings to reduce noise and improve accuracy.

python
Copy code

import numpy as np

class KalmanFilter:
    def __init__(self, process_noise, measurement_noise, estimation_error):
        self.process_noise = process_noise
        self.measurement_noise = measurement_noise
        self.estimation_error = estimation_error
        self.value = 0
        self.bias = 0

    def update(self, measured_value):
        kalman_gain = self.estimation_error / (self.estimation_error + self.measurement_noise)
        self.value += kalman_gain * (measured_value - self.value)
        self.estimation_error = (1 - kalman_gain) * self.estimation_error + abs(self.value) * self.process_noise
        return self.value

# Example usage
kf = KalmanFilter(process_noise=1e-5, measurement_noise=1e-2, estimation_error=1e-1)
filtered_value = kf.update(measured_value)

6. Motor Noise Mitigation
Adaptive Noise Cancellation
Use accelerometer and gyroscope data combined with motor control signals to filter noise adaptively.

python
Copy code

import numpy as np

def adaptive_noise_cancellation(imu_data, motor_signals):
    filtered_data = {}
    for axis in ["x", "y", "z"]:
        imu_signal = np.array([data[axis] for data in imu_data])
        motor_signal = np.array(motor_signals)
        noise = np.mean(motor_signal)
        filtered_data[axis] = imu_signal - noise
    return filtered_data

Final Integration
Real-Time Processing Loop
Combine all functionalities into a main loop.

python
Copy code

while True:
    # Collect data
    all_data = collect_all_sensor_data(i2c_addresses, i2c)
    
    # Apply calibration
    calibrated_data = {addr: apply_offsets(data["gyroscope"], gyro_offsets[addr]) for addr, data in all_data.items()}
    
    # Triangulate touch
    touch_locations = triangulate_touch(calibrated_data)
    print(f"Touch Locations: {touch_locations}")
    
    # Compute kinematics
    joint_angles = {joint: calculate_joint_angle(data) for joint, data in all_data.items()}  # Placeholder
    kinematic_positions = forward_kinematics(joint_angles, SKELETON)
    print(f"Kinematic Positions: {kinematic_positions}")
    
    # Store data
    log_touch_data(touch_locations, intensity=1.0)  # Placeholder for touch intensity
    log_kinematic_data(kinematic_positions)
    
    time.sleep(0.01)  # Adjust for real-time responsiveness

Next Steps
Integration Testing: Test each component with real hardware.
Performance Optimization: Minimize latency and optimize calculations.
Visualization: Implement a real-time 3D model of the skeleton for debugging.
Let me know if you'd like help implementing any specific part!

 

[amberwolf]

Windows python code to create 3d visual map of sensor data v1

I would definitely like the Windows-based PC code for a 3D visualization of the data from this system, along with the code to acquire and store that data from the RPI network.
ChatGPT said:
ChatGPT
Here’s how we can implement a Windows-based 3D visualization tool and a data acquisition/storage system for the wolf skeleton:

System Overview
Data Acquisition and Storage

Raspberry Pi network sends data to the PC via a TCP/IP or WebSocket connection.
Data includes touch triangulation, kinematic positions, and system timestamps.
3D Visualization

Use Python with PyOpenGL and Pygame or Unity for rendering a 3D skeleton.
Display real-time skeleton motion and touch events.
Data Flow

Data from Raspberry Pi nodes → Sent via sockets to PC → Stored locally → Visualized in 3D.
Step 1: Data Acquisition on PC
Server Code on Windows PC to Receive Data
python
Copy code

import socket
import json

# Server to receive data
HOST = '0.0.0.0'  # Listen on all interfaces
PORT = 5555       # Port number

def start_server():
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server_socket:
        server_socket.bind((HOST, PORT))
        server_socket.listen()
        print(f"Server listening on {HOST}:{PORT}")

        while True:
            conn, addr = server_socket.accept()
            with conn:
                print(f"Connected by {addr}")
                data = conn.recv(4096)
                if data:
                    process_data(data)

def process_data(data):
    try:
        # Assuming data is JSON
        parsed_data = json.loads(data.decode('utf-8'))
        print("Received Data:", parsed_data)
        store_data(parsed_data)
    except json.JSONDecodeError as e:
        print(f"Error decoding JSON: {e}")

def store_data(data):
    with open("skeleton_data.json", "a") as file:
        file.write(json.dumps(data) + "\n")

if __name__ == "__main__":
    start_server()

Step 2: Sending Data from Raspberry Pi
Here’s the Python client code for Raspberry Pi to send JSON data to the server:

python
Copy code

import socket
import json

HOST = '192.168.1.100'  # Replace with PC's IP address
PORT = 5555

def send_data(data):
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as client_socket:
        client_socket.connect((HOST, PORT))
        client_socket.sendall(json.dumps(data).encode('utf-8'))

# Example usage
data = {
    "timestamp": "2025-01-06T12:00:00",
    "touch_location": [10, 15, 20],
    "intensity": 0.75,
    "kinematic_positions": {"joint1": [0, 0, 0], "joint2": [5, 5, 5]},
}
send_data(data)

Step 3: 3D Visualization with PyOpenGL
Install Required Libraries
bash
Copy code

pip install pygame PyOpenGL numpy

Python 3D Visualization Code
This code renders the skeleton and updates positions/touches in real time.

python
Copy code

import pygame
from pygame.locals import *
from OpenGL.GL import *
from OpenGL.GLU import *
import socket
import threading
import json
import time

# Skeleton structure
SKELETON = {
    "joint1": {"position": [0, 0, 0], "children": ["joint2"]},
    "joint2": {"position": [5, 5, 5], "children": []},
}

# Global variables for real-time updates
kinematic_positions = {}
touches = []

def render_sphere(position, radius=0.2, color=(1, 0, 0)):
    glColor3f(*color)
    glPushMatrix()
    glTranslatef(*position)
    glutSolidSphere(radius, 20, 20)
    glPopMatrix()

def render_line(start, end, color=(1, 1, 1)):
    glColor3f(*color)
    glBegin(GL_LINES)
    glVertex3fv(start)
    glVertex3fv(end)
    glEnd()

def render_skeleton():
    for joint, data in SKELETON.items():
        position = kinematic_positions.get(joint, data["position"])
        render_sphere(position)
        for child in data["children"]:
            child_pos = kinematic_positions.get(child, SKELETON[child]["position"])
            render_line(position, child_pos)

def handle_touch_events():
    for touch in touches:
        render_sphere(touch, radius=0.5, color=(0, 1, 0))

def data_listener():
    global kinematic_positions, touches
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server_socket:
        server_socket.bind(('0.0.0.0', 5555))
        server_socket.listen()
        print("3D Viewer listening for data...")
        while True:
            conn, _ = server_socket.accept()
            with conn:
                data = conn.recv(4096)
                if data:
                    parsed_data = json.loads(data.decode('utf-8'))
                    kinematic_positions = parsed_data.get("kinematic_positions", {})
                    touches = [parsed_data.get("touch_location", [])]

def main():
    pygame.init()
    display = (800, 600)
    pygame.display.set_mode(display, DOUBLEBUF | OPENGL)
    gluPerspective(45, (display[0] / display[1]), 0.1, 50.0)
    glTranslatef(0.0, 0.0, -30)
    glEnable(GL_DEPTH_TEST)

    threading.Thread(target=data_listener, daemon=True).start()

    while True:
        for event in pygame.event.get():
            if event.type == pygame.QUIT:
                pygame.quit()
                quit()

        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
        render_skeleton()
        handle_touch_events()
        pygame.display.flip()
        pygame.time.wait(10)

if __name__ == "__main__":
    main()

Features
Skeleton Rendering: The code renders the skeleton as connected spheres and lines.
Touch Events: Highlights touch locations in green.
Real-Time Updates: Listens for incoming data from Raspberry Pi and updates the 3D visualization dynamically.
Next Steps
Enhance Visualization: Add labels, joint rotation, or motion trails.
Optimize Performance: Batch updates for smoother rendering.
Advanced Graphics: Consider integrating Unity or Unreal Engine for more advanced visuals. Data can be streamed using WebSocket or REST APIs.
Data Analysis: Integrate touch pressure mapping and motion analytics.
Let me know how you'd like to refine or extend this functionality!

 

[amberwolf]

Windows python code to enhance 3d visual map of sensor data v1

Thank you, yes, I would like the code for those next steps.
ChatGPT said:
ChatGPT
Here’s how we can implement enhanced visualization, performance optimization, advanced graphics with Unity, and data analytics for your robotic wolf skeleton:

Next Steps Implementation
1. Enhanced Visualization
Add Joint Labels and Motion Trails
Update the PyOpenGL Code:

Include text labels for each joint.
Implement motion trails using a history buffer.
python
Copy code

from OpenGL.GLUT import glutBitmapCharacter, GLUT_BITMAP_HELVETICA_18
from collections import deque

# Store motion trails for joints
motion_trails = {joint: deque(maxlen=50) for joint in SKELETON.keys()}

def render_label(position, text):
    glColor3f(1, 1, 0)
    glRasterPos3f(position[0], position[1], position[2])
    for char in text:
        glutBitmapCharacter(GLUT_BITMAP_HELVETICA_18, ord(char))

def render_skeleton_with_labels():
    for joint, data in SKELETON.items():
        position = kinematic_positions.get(joint, data["position"])
        render_sphere(position)
        render_label(position, joint)

        motion_trails[joint].append(position)
        render_motion_trail(motion_trails[joint])

        for child in data["children"]:
            child_pos = kinematic_positions.get(child, SKELETON[child]["position"])
            render_line(position, child_pos)

def render_motion_trail(trail, color=(0, 0.5, 1)):
    glColor3f(*color)
    glBegin(GL_LINE_STRIP)
    for point in trail:
        glVertex3fv(point)
    glEnd()

# Replace render_skeleton with render_skeleton_with_labels in the main loop.

2. Performance Optimization
Batch Updates
Optimize rendering and data processing by batching updates.

python
Copy code

def batch_process_data(data_list):
    global kinematic_positions, touches
    for data in data_list:
        kinematic_positions.update(data.get("kinematic_positions", {}))
        if "touch_location" in data:
            touches.append(data["touch_location"])

# Modify the listener to handle batched data
def data_listener():
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server_socket:
        server_socket.bind(('0.0.0.0', 5555))
        server_socket.listen()
        print("3D Viewer listening for data...")
        while True:
            conn, _ = server_socket.accept()
            with conn:
                data = conn.recv(4096)
                if data:
                    parsed_data = json.loads(data.decode('utf-8'))
                    if isinstance(parsed_data, list):
                        batch_process_data(parsed_data)
                    else:
                        process_data(parsed_data)

3. Advanced Graphics with Unity
Step 1: Unity Project Setup
Install Unity Hub and create a new 3D project.
Import Socket.IO package (for WebSocket communication) or write a TCP client.
Step 2: Import 3D Model
Create a 3D wolf skeleton model in Blender or Maya and import it into Unity.
Set up bones and rigging for animation.
Step 3: Unity Script for Data Integration
csharp
Copy code

using System;
using System.Net.Sockets;
using System.Text;
using UnityEngine;

public class IMUVisualization : MonoBehaviour
{
    public GameObject[] Joints; // Assign joints in Unity Editor
    private TcpClient client;
    private NetworkStream stream;

    void Start()
    {
        ConnectToServer("192.168.1.100", 5555);
    }

    void ConnectToServer(string host, int port)
    {
        try
        {
            client = new TcpClient(host, port);
            stream = client.GetStream();
            Debug.Log("Connected to server");
        }
        catch (Exception e)
        {
            Debug.LogError("Connection error: " + e.Message);
        }
    }

    void Update()
    {
        if (stream != null && stream.DataAvailable)
        {
            byte[] buffer = new byte[4096];
            int bytesRead = stream.Read(buffer, 0, buffer.Length);
            string data = Encoding.UTF8.GetString(buffer, 0, bytesRead);
            ProcessData(data);
        }
    }

    void ProcessData(string data)
    {
        try
        {
            var parsedData = JsonUtility.FromJson<SkeletonData>(data);
            for (int i = 0; i < Joints.Length; i++)
            {
                Joints[i].transform.position = new Vector3(
                    parsedData.kinematic_positions[i].x,
                    parsedData.kinematic_positions[i].y,
                    parsedData.kinematic_positions[i].z
                );
            }
        }
        catch (Exception e)
        {
            Debug.LogError("Data processing error: " + e.Message);
        }
    }
}

[Serializable]
public class SkeletonData
{
    public JointPosition[] kinematic_positions;
}

[Serializable]
public class JointPosition
{
    public float x;
    public float y;
    public float z;
}

Step 4: Real-Time Visualization
Use Unity’s animation tools to visualize touch interactions and skeleton motion.
Add particle effects for touch locations.
4. Data Analytics
Add Touch Pressure Mapping
Use a heatmap to visualize touch intensity over time.

python
Copy code

import matplotlib.pyplot as plt
import numpy as np

# Heatmap initialization
touch_map = np.zeros((100, 100))

def update_touch_map(location, intensity):
    x, y = int(location[0]), int(location[1])
    touch_map[x, y] += intensity

def display_heatmap():
    plt.imshow(touch_map, cmap="hot", interpolation="nearest")
    plt.colorbar()
    plt.show()

# Call update_touch_map for each touch and periodically display_heatmap.

Next Steps for You
Unity Implementation: Set up Unity for advanced 3D visualization with real-time animations.
Data Analytics: Expand heatmaps to include time-series analysis (e.g., rate of touches).
System Testing: Test integration between Raspberry Pi, server, and visualization tools.
Let me know where you’d like to dive deeper or need assistance!

 

[amberwolf]

Windows python code to optimize and animate 3d visual map of sensor data v1

Yes I would like these next steps as well.
ChatGPT said:
ChatGPT
Here’s a detailed implementation of the next steps, expanding on Unity's visualization capabilities, advanced analytics, and system integration testing. Each step will provide clear directions, scripts, and concepts to ensure seamless functionality.

Advanced Steps
1. Unity: Enhanced Real-Time Visualization
1.1 Touch Effects and Heatmap Integration
Goal: Add visual feedback in Unity for detected touches, including intensity-based particle effects and real-time heatmap overlays.

Unity Script for Particle Effects: Attach this script to each joint's GameObject.

csharp
Copy code

using UnityEngine;

public class TouchEffect : MonoBehaviour
{
    public ParticleSystem touchParticles; // Assign a ParticleSystem in Unity Editor

    public void TriggerTouchEffect(float intensity)
    {
        var main = touchParticles.main;
        main.startSize = Mathf.Clamp(intensity, 0.1f, 1.0f);
        touchParticles.Play();
    }
}
Modify IMUVisualization Script to Trigger Touch Effects:

csharp
Copy code
void ProcessData(string data)
{
    try
    {
        var parsedData = JsonUtility.FromJson<SkeletonData>(data);
        
        // Update joint positions
        for (int i = 0; i < Joints.Length; i++)
        {
            Joints[i].transform.position = new Vector3(
                parsedData.kinematic_positions[i].x,
                parsedData.kinematic_positions[i].y,
                parsedData.kinematic_positions[i].z
            );
        }

        // Trigger touch effects
        if (parsedData.touch_location != null)
        {
            int jointIndex = FindClosestJoint(parsedData.touch_location);
            float intensity = parsedData.touch_intensity;

            Joints[jointIndex].GetComponent<TouchEffect>().TriggerTouchEffect(intensity);
        }
    }
    catch (Exception e)
    {
        Debug.LogError("Data processing error: " + e.Message);
    }
}

int FindClosestJoint(Vector3 touchLocation)
{
    int closestJoint = 0;
    float minDistance = float.MaxValue;

    for (int i = 0; i < Joints.Length; i++)
    {
        float distance = Vector3.Distance(touchLocation, Joints[i].transform.position);
        if (distance < minDistance)
        {
            minDistance = distance;
            closestJoint = i;
        }
    }
    return closestJoint;
}

1.2 Heatmap Overlay
Heatmaps are dynamically rendered on the robot's surface to visualize touch intensity.

Unity Shader for Heatmap Visualization
Create a custom shader to blend touch intensity on the skeleton’s surface.

Shader Code (HeatmapShader.shader):

glsl
Copy code

Shader "Custom/HeatmapShader"
{
    Properties
    {
        _MainTex ("Texture", 2D) = "white" {}
        _Heatmap ("Heatmap Texture", 2D) = "white" {}
    }
    SubShader
    {
        Tags { "RenderType"="Opaque" }
        LOD 200

        CGPROGRAM
        #pragma surface surf Lambert

        sampler2D _MainTex;
        sampler2D _Heatmap;

        struct Input
        {
            float2 uv_MainTex;
            float2 uv_Heatmap;
        };

        void surf(Input IN, inout SurfaceOutput o)
        {
            fixed4 mainTex = tex2D(_MainTex, IN.uv_MainTex);
            fixed4 heatTex = tex2D(_Heatmap, IN.uv_Heatmap);
            o.Albedo = mainTex.rgb * (1 + heatTex.r); // Amplify color by heatmap intensity
        }
        ENDCG
    }
    FallBack "Diffuse"
}

Unity Script to Update Heatmap Texture: Generate a texture dynamically based on incoming touch data.

csharp
Copy code

using UnityEngine;

public class HeatmapGenerator : MonoBehaviour
{
    public Texture2D heatmapTexture;
    public Renderer targetRenderer;

    void Start()
    {
        heatmapTexture = new Texture2D(256, 256);
        targetRenderer.material.SetTexture("_Heatmap", heatmapTexture);
    }

    public void UpdateHeatmap(Vector2 touchPoint, float intensity)
    {
        int x = Mathf.Clamp((int)(touchPoint.x * heatmapTexture.width), 0, heatmapTexture.width - 1);
        int y = Mathf.Clamp((int)(touchPoint.y * heatmapTexture.height), 0, heatmapTexture.height - 1);

        Color currentColor = heatmapTexture.GetPixel(x, y);
        heatmapTexture.SetPixel(x, y, currentColor + new Color(intensity, 0, 0));
        heatmapTexture.Apply();
    }
}

2. Advanced Analytics
2.1 Predictive Touch Mapping
Train a neural network to predict touch pressure and locations for missing or noisy data.

Python Implementation Using TensorFlow:

python
Copy code

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout

# Load preprocessed data
def load_data():
    # Replace with real data loading
    return train_data, train_labels, test_data, test_labels

train_data, train_labels, test_data, test_labels = load_data()

# Build the model
model = Sequential([
    Dense(128, activation='relu', input_shape=(train_data.shape[1],)),
    Dropout(0.2),
    Dense(64, activation='relu'),
    Dense(3, activation='linear')  # Predict x, y, z
])

model.compile(optimizer='adam', loss='mse', metrics=['mae'])
model.fit(train_data, train_labels, epochs=10, validation_data=(test_data, test_labels))

# Save model for deployment
model.save("touch_predictor_model.h5")

2.2 Time-Series Analysis for Motion
Use Python to analyze and predict skeleton movements over time.

python
Copy code

import pandas as pd
import matplotlib.pyplot as plt

# Load motion data
data = pd.read_csv("skeleton_data.csv")

# Plot motion of specific joint
joint = "joint1"
plt.plot(data["timestamp"], data[f"{joint}_x"], label="X-axis")
plt.plot(data["timestamp"], data[f"{joint}_y"], label="Y-axis")
plt.plot(data["timestamp"], data[f"{joint}_z"], label="Z-axis")
plt.legend()
plt.show()

3. System Integration Testing
3.1 Automated Testing
Write Python scripts to simulate touch and motion data to test the entire pipeline.

python
Copy code

import socket
import json
import time
import random

HOST = '192.168.1.100'  # Replace with PC IP
PORT = 5555

def simulate_data():
    data = {
        "timestamp": time.time(),
        "touch_location": [random.uniform(0, 10), random.uniform(0, 10), random.uniform(0, 10)],
        "touch_intensity": random.uniform(0.1, 1.0),
        "kinematic_positions": {f"joint{i}": [random.uniform(0, 10), random.uniform(0, 10), random.uniform(0, 10)] for i in range(10)}
    }
    return data

def send_data(data):
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as client_socket:
        client_socket.connect((HOST, PORT))
        client_socket.sendall(json.dumps(data).encode('utf-8'))

if __name__ == "__main__":
    while True:
        data = simulate_data()
        send_data(data)
        time.sleep(0.1)

Next Steps for Development
Deploy the Heatmap Shader: Test in Unity with real-time input.
Train Predictive Models: Use the touch dataset for machine learning.
Integrate Advanced Analytics: Plot touch patterns and predict missing data.
Optimize Unity Rendering: Use GPU instancing for high performance.
Let me know how you’d like to proceed or if specific features require further expansion!

 

[amberwolf]

Process and code snips to train AI responses v1

I will need to use this touch and position data to train an AI model so that it responds like a wolf or dog to user input, with realistic-wolf-like motion control of the motor system to position and move the skeleton (and other body parts).

ChatGPT said:
ChatGPT
To create a system where the robotic wolf skeleton responds to user input with realistic, wolf-like behavior, we can design a pipeline that includes data collection, model training, and real-time deployment of a neural network for behavior generation. Here's how we can proceed:

Steps to Build the AI Model for Wolf-Like Behavior
1. Collecting Data for AI Training
1.1 Recording Touch and Position Data
Touch and Position Dataset:

Continuously record touch intensity and location alongside the robot's current kinematic state.
Store the motor control signals (commands sent to actuators) for each state to understand how the robot moved in response.
Example Schema for Training Dataset:

json
Copy code

{
    "timestamp": 1672619900.123,
    "touch": {
        "location": [1.2, 0.5, 0.7],
        "intensity": 0.8
    },
    "kinematics": {
        "joint1": [0.5, 0.1, 0.2],
        "joint2": [0.6, 0.3, 0.4],
        ...
    },
    "motor_commands": {
        "joint1": 0.4,
        "joint2": -0.2,
        ...
    }
}

Simulated or Manually Designed Behavior:

Define a set of pre-programmed behaviors based on touch intensity and location (e.g., lifting a paw, turning the head, wagging the tail).
Generate data by manually controlling the robot to simulate realistic responses to touch.
2. Training the AI Model
2.1 Choosing the Model Architecture
Use a Recurrent Neural Network (RNN) or a Transformer model to learn temporal dependencies from the data.

2.2 Model Input and Output
Input: Touch location, intensity, and kinematic positions of joints.
Output: Motor control commands for joints and actuators.
2.3 Python Code for Training the Model
python
Copy code

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, LSTM, Input, Dropout

# Load dataset
def load_data():
    # Replace with code to load your touch and motor data
    return train_data, train_labels, test_data, test_labels

train_data, train_labels, test_data, test_labels = load_data()

# Build the model
model = Sequential([
    Input(shape=(None, train_data.shape[2])),  # Time-series input
    LSTM(128, return_sequences=True, activation='relu'),
    Dropout(0.2),
    LSTM(64, activation='relu'),
    Dense(train_labels.shape[2], activation='tanh')  # Output motor commands
])

model.compile(optimizer='adam', loss='mse', metrics=['mae'])

# Train the model
model.fit(train_data, train_labels, epochs=50, batch_size=32, validation_split=0.2)

# Save the trained model
model.save("wolf_motion_model.h5")

3. Real-Time Deployment
3.1 Loading the Model for Deployment
Use TensorFlow Lite for faster inference on embedded systems like Raspberry Pi.

python
Copy code

import tensorflow as tf
import numpy as np

# Load the trained model
model = tf.keras.models.load_model("wolf_motion_model.h5")

# Real-time input from sensors
def process_real_time_data(touch_data, kinematics_data):
    input_data = np.expand_dims(np.array([touch_data + kinematics_data]), axis=0)
    motor_commands = model.predict(input_data)
    return motor_commands

4. Generating Wolf-Like Motion
4.1 Designing the Motion Control System
Map the predicted motor commands to physical actuators.
Implement safety checks to avoid unnatural joint positions or collisions.
4.2 Motion Blending for Smooth Transitions
Use Bezier curves or interpolation to ensure fluid movement between states.

python
Copy code

def interpolate_motion(start_pos, end_pos, steps=10):
    motion_path = []
    for t in range(steps):
        ratio = t / float(steps - 1)
        interpolated_pos = start_pos * (1 - ratio) + end_pos * ratio
        motion_path.append(interpolated_pos)
    return motion_path

5. Behavioral AI: Responding to Input Like a Wolf
5.1 Emotion and Behavior State Machine
Introduce emotional states (e.g., curious, aggressive, calm) that influence behavior.

State Machine Example:

python
Copy code

class WolfStateMachine:
    def __init__(self):
        self.state = "calm"  # Initial state

    def update_state(self, touch_data):
        if touch_data["intensity"] > 0.8:
            self.state = "aggressive"
        elif touch_data["location"][1] > 0.7:  # Head area
            self.state = "curious"
        else:
            self.state = "calm"

    def get_motor_commands(self, touch_data, kinematics_data):
        if self.state == "aggressive":
            return aggressive_response(touch_data, kinematics_data)
        elif self.state == "curious":
            return curious_response(touch_data, kinematics_data)
        else:
            return calm_response(touch_data, kinematics_data)

6. Visualization of Behavior
6.1 Display Real-Time States
Update the Unity visualization to include behavior state overlays (e.g., color-changing fur for emotion).
6.2 Record and Analyze Behavior
Log motion and behavior states for further refinement.
7. Testing and Refinement
Simulation in Unity: Test the trained model in a simulated Unity environment to verify motion accuracy.
Real-World Testing: Deploy the model on the robotic skeleton and iteratively refine it based on observed behavior.
Noise Filtering: Apply additional filtering techniques to handle noisy sensor data in real-time.
Next Steps
Expand the Dataset: Collect diverse touch-response scenarios for robust learning.
Train Advanced Models: Experiment with Reinforcement Learning for adaptive behavior.
Integration: Combine the behavior AI with advanced visualization and analytics systems.
Let me know if you'd like the detailed implementation of specific components or help in setting up any part of the pipeline!

 

[amberwolf]

Thats it for today's ChatGPT stuff. (ran out of free access) When I get a chance I'll try testing some of the above stuff to see if it will do anything (much less what it is supposed to do). And then if not, how it is supposed to work so I can figure out what's broken to fix it. (that will probably take a long time).

EDIT: BTW, I was wrong about it not providing code at all; it did do it at least once back in Dec 2023. I don't recall if the code actually worked or not (or if I even tried it, since it was just giving me a guide to get started); that's back on the first page of this thread.

I thought that it kept everything in the chat history, but there are things I tried out that are not there. I don't recall the details of what they were, but they probably didn't have useful results (like most of the interactions I've had with it and other similar AI pages, where the results *sound* reasonable, but....).

 

[mxlemming]

You have to remember when using this that it's just machine generated code and there's little point saving it. It just fills your thread with garbage.

It becomes valuable only once you've tested it, debugged it, integrated...

 

[amberwolf]

I'm sure you're right but if I save it here then it's there for me to access, in case I have a local system problem that loses the data (since the actual ChatGPT page doesn't apparently save all of the chats). Gives me a starting point at least, since I don't really know any of this coding language yet (trying to learn, but while I understand the process, my brain doesn't work the way it needs to to really "get" the actual programming language (or any other language), etc. I don't learn things in a linear sequence either--I do it in jumps and leaps and back-and-forth bits, and there are always parts of things that remain beyond my grasp because my chesspiece keeps jumping over that square and can't land on it. It's hard to describe the actual problem, but it's why all this is taking so long for me.

Machine generated code is better than anything I can create.

I should probably make two other separate threads for this--one to stick all the discussion posts in, and one for all the bits and bots like this stuff, leaving this one only for the actual stuff that is happenign and the subsystem descriptions, etc. I didn't do that before because I didn't want to clutter up the forums like that, but it might make it easier for people coming into the thread to figure out what's going on.