Robotics Project: Snuggles, The Wolf

[amberwolf]

There's a lot of separate things and a lot of words in my responses so I split them up in to more than one post below:

There's really no need to use modeling for the position of limbs. I guess I left out that neural networks are incredibly slow when compared to old school methods, like formulas and equations. Even if the hardware were able to keep up, the video card in my desktop uses up to 350 watts of electricity to do what it does -- which is more than half the average watts of my ebike (500).

Ok. I don't yet know how to do this stuff, or which things are good for what. So whatever alternatives you (or anyone else) can think of for processing this data, I'm up for trying out.

Granted, you could have a pretty big battery in a St. Bernard sized body, but it wouldn't last much longer than a short nap I don't think.

Well, the computer(s) won't be in the robot itself (they'd probalby catch fire inside the heat-insulating furry body!***), and since this version of the project is essentially for a relatively stationary robot (imagine a big old tired lazy dog that just lays on the bed and waits for you to get home, then snuggles, plays, etc., all just right there in one place or limited area, it doens't need to be battery powered either.

Since I would like all the noisy motors and stuff outside the robot wherever possible, I'm already going to have a (probably large) umbilical of push/pull cables from skeleton/etc out to the actuators, and so power for any of the internal electronics (sensors, first-level MCU processing, speakers, etc). would come via a cable within that bundle, too.



Later I'd love to develop something that could actually follow me around or even be portable and "self powered", but I dont' actually need that.


***In the 90s I worked on desiging a wolf-shaped computer case, but I couldn't come up with anything other than liquid cooling that passed the refrigeration tubes out to something large and external that would let the computer inside function for more than a few minutes; IIRC it was a dual-P90 with WinNT when I started, I think I'd moved to Win2000 and some newer CPU/etc by the time I concluded it just wouldn't work without the external stuff that would be about as big and noisy as a regular computer tower, and dropped the idea, since all this was for my music-recording home-studio stuff, and I already could easily make a regular computer case nearly silent.

 

[amberwolf]

Possibly. Off hand I'm not really sure, to be honest; I don't think I'm following you very well. It doesn't have to walk though, so I still think your best bet is just using simple formulas and not deep learning. I'm talking about controlling the servos, e.g. to explicitly position a limb in an arbitrary position or something; deciding where to position the limb is another matter altogether.

I might have to make a 3D model of the idea, and animate it, with narration, to explain it. I can clearly see this in my head...maybe if I open it iup and take pictures? :lol:



Before I can even get to the point of creating a command to control the servos, I have to know where the interactions are. Figuring that out is not always as easy as "hey sensor 1 on paw 1 got a certain amount of input"...some of that input is on that paw, and some of it might also be detected in other paws.

To know where those are, I have to know where the parts are, relative to each other, and knowing that I can map that out in a 3D space.

Then all the vibration/etc sensory input, which is going to have inputs from multiple sensors on different portions of a body part, or even separate body parts, detecting just one interaction, so the system has to figure out where the center of that came from, like an earthquake epicenter, etc. can be placed on that map since the machine already knows where each sensor is if it knows where the body part is that it's attached to.

When the location(s) of interaction has been determined, then the proper response can be determined.

There are simplistic ways of just ignoring a bunch of data and assuming a response comes only from one spot, etc., but this will result in incorrect responses in various instances where those aren't the case.


Maybe there's some less complicated way of doing all this but I've been trying to find one for a long time and it just doesn't appear to exist.

The only thing I have found that could be used to localize touches over a body is IMUs (or accelerometers) because they can detect very small vibrations, *and* they can tell which direction they're in...but since they can't tell *where* they are, data from multiple sensors have to be integrated togehter to figure that out. (there are probably other ways than a mapping function to do that...but the mapping function just seems like the best way since you get a lot more out of it than just the simple touch data, and it could then be used to do other things with).


It can't use pressure sensors to determine touches, pets, etc., because it's almost all covered in fur. A touch on that won't register on any such sensors that are under the fur, you'd have to mash it hard enough to push the fur all the way down and be mashing the "skin" instead, and that's not that much of how people interact with dogs (or other fuzzy beings).

Even if it did use pressure sensors for this, it would require them over the entire "skin" of the body, and I don't know of anything that could be built that way--there's resistive cloth that can do some forms of touch detection, but AFAICT it wouldn't work under the fur very well, if at all.

Capacitive touch sensors can't be close enough together to do this, and it would require far too many of them (hundreds or more), and none of the ones I coudl afford that many of can do anything more than on/off detection, *and* they mostly don't work beyond about 1cm in open air and (far) less thru materials.

In theory a theremin-like radio-antenna system could work thru the fur and detect analog signals and work over a relatively large surface area...but not in such close proximity to each other, without some fairly robust radio-engineering I'm not capable of to prevent interference with each other.

Radar sensors that I coudl afford only do on/off detection and only more or less one per room, as they also interfere with each other. Not remotely useful for smallscale touch detection.

If you (or anyone else) knows of sensor systems (that I coudl actually afford on a nonexistent budget :lol: ) that could detect light touches anywhere on a body with only a "few" (probably at least a couple of dozen) of them, and still be used to localize the interactions to specific spots on the body (not just at the sensors), I'd love to know about them so I can investigate.

 

[amberwolf]

I'm not sure what issues you're currently trying to solve? Maybe I'm just not following you at all, but here's how I see it:

1. Actuators generally report their position, don't they? So I'm not sure what problem needs to be addressed there?

Yes, and no. Depends on the actuator you use and whether it is one that has to be direclty monitored by the software.

Servos (probably most of what I'd be using for actuators) have a built in feedback to their internal driving electronics. You just feed them a PWM signal at a specific duty cycle (and frequency) and the move to the angle you told them to. (there are 360 degree servos that don't have the feedback so you have to build your own position sensor into the mechanism to tell where it's at, and code for that)).

So...you know where you told them to be, but you don't know where they actually are. If there is sufficient resistance against them, they can't move ot where you told them to be, so they'll be somewhere between that and their "off" point. To know where the controlled object is actually located, you have to have a sensor in the object itself that can tell you that--that's part of what the IMUs would do.

All position sensors are relative anyway, whenever they are attached to other moving parts. They only tell you where a part is from it's "zero" position. They don't tell you where it is in relation to it's environment, or to some other part that it's attached to that may have moved or even be in motion.

For instance, just a plain simplistic "robot arm" that's fixed to a tabletop, and has a 3-axis shoulder, one-axis elbow, two-axis wrist, and one-axis two-finger-gripper-hand. The hand sensors know if it's open or closed, but it doesn't know if it's pointing up or down or left or right,--that's up to the wrist. The wrist knows it's up, down, left, or right, but it doesn't know if it's at an angle; that's up to the elbow. The elbow knows what angle it's at, but not which direction it's pointing; that's up to the shoulder. The shoulder can move in all three axes to point the entire assembly in any direction above the table (so 180 degrees of freedom for two axes, and 360 for the last).

So the data of each sensor set further down the line is dependent on the data of each previous sensor data.

A robot that is not fixed in a particular spot relative to it's environment, and has many more degrees of freedom, has a whole lot more dependencies between the sensor data sets to figure out where the heck any particular part of the robot actually is, and whether it is likely to interact with some portion of itself or it's environment.

That's where the wolfy-bot is at. It's not as bad a problem as one that has to walk around...but it's still pretty complicated.



EDIT (hopefully this is comprehensible, I woke from a(nother) "processing dream" about this (I have lots of these, in between the emotionally-driven nightmares about stressful events past and present that are much of the reason for needing this project to succeed):

I forgot to also state (in this reply; I think it's covered in ohter posts before, though) that in the wolfy, it's likely that none of the actuators will be inside the actual robot, but will be outside in a box to mute their sounds (etc), and will cable-operate their components. So there will be slop in the response that the actualtor's direct sensing can't know about. To get correct info it needs to know where the actual components being driven are, and for that it needs data from the IMUs attached to the parts.

This also lets it know if something has broken between the actuators and the components they drive, as the component wouldnt' be moving as the actuator is, and the difference in response from expectation would help it know that and allow a response to this to alert the user.

In addition (as also I think covered previously), the user interactions are going to include the user moving the robot in whole or in part, and it has to "know" this is happening to "cooperate" with it for the most part, or to "resist" it if playfulness responses indicate it should for that situation, etc. The IMU data would show what's actually happening to the robot's components to let it do this.


And I probably wasn't clear before on the human-readable data output the map would generate. Having such a visible 3d model (regardless of primitivity of actual display) would make testing and troubleshooting of the sensory system itself, which is going to be necessarily complex in operation. It would allow visualization of the touches, to see that it is correctly detecting where the touches are and their intensities and durations, so that localization and interpretation of the data could be corrected if it isn't, during initial development.

Beyond ID it would be a useful testing tool to be sure sensors are operating as they were during training, dev, etc. since it would be visually obvious that they've changed outputs without sorting thru a bunch of signals, data, and code, or disassembling hardware (once assembled, it is not easy to get inside for testing, so it would be very good to have a visual representation of where a problem probably lies to then check the data sources to then know where to open it up).

Iv'e done lots of hardware troubleshooting over my life, and a visual tool like this would be invaluable for these purposes, totally aside from the 3d map's usefulness in localizing the touches for more correct behavioral responses.

 

[amberwolf]

2. Detecting interaction with the environment is a tricky problem, as is processing that data and deciding what position the actuators should be put in. It sounds like you're trying to have it figure everything out every time it gets turned on. I think a much simpler, much more effective way of doing it is to impose certain constraints on the entire apparatus -- as in make assumptions about the design. A pressure sensor on the right foot will always send a signal that is from the right foot, and the place of attachment of that right limb can be assumed to always be to the right of the left foot. So there's no reason to detect whether or not the right foot is to the left of the left foot, or to the right of it (I assume the limbs are going to be too simple to have that much range of motion).

I guess I dont' really know how to explain it, because it isn't designed the way you're describing. It doesn't have pressure sensors, it's using the IMUs (acclerometer/gyros). As noted in the part of the reply just above this, data can come into multiple sensors and you have to know where they are relative to each other to know what the data means / where it came from, in some cases. (not for all cases...it is about how a dog senses and reacts to interactions (for this discussion, it's specifically about touch). If necessary I can try to write a scenario list of interactions and responses; that's what I expected to have to do to create if/then/etc lists for the program to interpret the data, but it is going to take a long while to do).

I'm not sure what you need all these sensors for, and if you had all of this data, I'm not sure what you'd want to do with it.

Hopefully the explanations above will help with that (not sure since I thought I'd basically explained it earlier in the thread, but I'll have to re-read what I've already posted to see if I actually did that properly).

If it doesn't help, I'll try to come up with better explanations and descriptions somehow.

 

[amberwolf]

Robotics isn't my usual area, but I can at least estimate whether or not something would work well when applying machine learning to it -- and I honestly can't say whether or not all of that data could be processed with neural networks. I'm having too much difficulty trying to evaluate the setup. I'd need more familiarity with the components and the output they produce, etc., as well as some concrete information about what output would be required from the machine learning part of the system.

Well, if NNs aren't the thing to process the data with, that's ok. It just sounded like they might make it easier based on the little I've learned so far about them, and the things you'd said about using it to process the other project's data you were working on.

The output of IMUs is, at essentials, XYZ axis data for gyroscope and accelerometer, for each sensor board.

Generally the data is in the numeric format of X.XX Y.YY Z.ZZ. (it might have more decimal places depending on sensitivity).

Let's say you're only reading the AM data and the board is sitting there not moving with no vibrations or anything, and you "calibrate" it so that is the zero position, you get zeros for all those 0.00 0.00 0.00. If you pick the board up and it stays perfectly level and not rotated at all, the vertical axis AM data will spike and keep changing until it stops moving and you then get zero again. If you're rotating it, then the AM data for all axes in rotation change to whatever acceleration that causes until you s top moving it, then you get zero again.

The gyro data will be zeros at the calibrated zero position (usually done at powerup to give you a known reference), then if you tilt the board the axes being tilted tell you the angle it's tilted at (depending on sensor that might be in degrees, or some other number you have to convert). If you tilt it only around the Z axis till that is up and down instead of horizontal, then you'd get 0.00 0.00 0.90 if it's in degrees. Etc. It stays at that reading until you move it back.



For the specific boards I happen to have, it can get more complicated if you use the onboard MPU to process things but I can't find much info on how to use that to do anything; the chip datasheets/etc don't include that, just the registers used to pass data back and forth--not what the data is, or what it tells the MPU to do...so I'm ignoring the MPU entirely since the raw gyro and AM data is fine.

 

[amberwolf]

I hope I got all that together in the right way...if anything doesn't make sense, point it out and I'll try to fix it.

 

[riding_on]

Since I would like all the noisy motors and stuff outside the robot wherever possible, I'm already going to have a (probably large) umbilical of push/pull cables from skeleton/etc out to the actuators, and so power for any of the internal electronics (sensors, first-level MCU processing, speakers, etc). would come via a cable within that bundle, too.

Okay, well that changes things pretty radically then: in that case, sure, you can go hog wild with the computational requirements (so long as you don't mind the possibility of being electrocuted in your sleep because you drooled on the AC plug, lol).

There's a whole lot here to digest, and I'll do my best to get to it all. I've been working 60+ hour weeks again for the past couple of months, so please don't take offense if I seem slow to respond. This is one of three links I try to check daily, so I'll get to it

 

[riding_on]

Yes, and no. Depends on the actuator you use and whether it is one that has to be direclty monitored by the software.

Servos (probably most of what I'd be using for actuators) have a built in feedback to their internal driving electronics. You just feed them a PWM signal at a specific duty cycle (and frequency) and the move to the angle you told them to. (there are 360 degree servos that don't have the feedback so you have to build your own position sensor into the mechanism to tell where it's at, and code for that)).

So...you know where you told them to be, but you don't know where they actually are. If there is sufficient resistance against them, they can't move ot where you told them to be, so they'll be somewhere between that and their "off" point. To know where the controlled object is actually located, you have to have a sensor in the object itself that can tell you that--that's part of what the IMUs would do.

You mean it's not possible to monitor the current and use simple formulas to judge whether or not it's having difficulty, or at a midpoint or something?

... I knew I should have taken at least an intro to electrical engineering course in school. I'm speechless.

 

[riding_on]

Hopefully the explanations above will help with that (not sure since I thought I'd basically explained it earlier in the thread, but I'll have to re-read what I've already posted to see if I actually did that properly).

If it doesn't help, I'll try to come up with better explanations and descriptions somehow.

I think our problem is primarily that all of your experience is in stuff that I have pretty much none in, and vice versa; definitely not your fault, I hope I didn't sound like I was implying that or anything.

Basically I need to "see" the raw data, or at least be able to imagine the 0s and 1s that would be fed into an artificial neural network, to have any more informed comments to add.

 

[riding_on]

Generally the data is in the numeric format of X.XX Y.YY Z.ZZ. (it might have more decimal places depending on sensitivity).

Let's say you're only reading the AM data and the board is sitting there not moving with no vibrations or anything, and you "calibrate" it so that is the zero position, you get zeros for all those 0.00 0.00 0.00. If you pick the board up and it stays perfectly level and not rotated at all, the vertical axis AM data will spike and keep changing until it stops moving and you then get zero again. If you're rotating it, then the AM data for all axes in rotation change to whatever acceleration that causes until you s top moving it, then you get zero again.

The gyro data will be zeros at the calibrated zero position (usually done at powerup to give you a known reference), then if you tilt the board the axes being tilted tell you the angle it's tilted at (depending on sensor that might be in degrees, or some other number you have to convert). If you tilt it only around the Z axis till that is up and down instead of horizontal, then you'd get 0.00 0.00 0.90 if it's in degrees. Etc. It stays at that reading until you move it back.

Here's where I'm currently tripping over having more to say: I don't know of a good way to process a live, constant data stream like that. It's not something I've read much of any research about. Usually these things just sit there and do nothing until you come along and prompt them to do something by feeding a discrete chunk of data to them. A continuous stream of data is a little different, and I'm not having much luck coming up with a practical way of handling it.

A naive approach (which is often surprisingly effective, don't get me wrong) would be to just buffer the data for periods of time -- say 5 seconds at a time or something -- and then process that 5 second chunk before repeating the process over and over again. I wanted to have something more interesting to say than that, but it hasn't come to me yet.

Well... I guess, actually, you could do it with a standard LSTM (long short-term memory) model. It would just mean feeding timesteps to it piece by piece. Yeah, nevermind, I'm just being stupid -- that would work fine. Could even have some convolutions for good measure if you wanted to try them out.

In my experience, designing an architecture to suit a problem is at least 50% trial and error. It's extremely difficult to know what would work best for a given application. It would be great if there were one model that was best for a big range of problems, but that's just not how it ends up working out in practice. You don't have to take my word for it, you can just go to paperswithcode.com and see the new architecture that's great for this-or-that on a daily basis. There's usually some credible sounding theory underlying them, but if you ask me, it really is just guessing and trying random stuff half the time to see what works best.

 

[amberwolf]

Okay, well that changes things pretty radically then: in that case, sure, you can go hog wild with the computational requirements (so long as you don't mind the possibility of being electrocuted in your sleep because you drooled on the AC plug, lol).

Not much chance of that; the power in would all be low-current 5VDC stuff (maybe 12V to power the speaker better, and use that to convert to 5v/etc internally, but unlikely to be necessary: at present I'm successfully using a cheap 5v amplifier PAM8403

https://www.aliexpress.us/item/2251832360726451.html?spm=a2g0o.order_detail.order_detail_item.7.7661f19cZU2wzs&gatewayAdapt=glo2usa

and sound-playback board DY-SV5W

https://www.aliexpress.us/item/3256801527174158.html?spm=a2g0o.order_detail.order_detail_item.9.7661f19cZU2wzs&gatewayAdapt=glo2usa

with a simple switch on some of the input pins to either snore or slowly quietly breathe as if sleeping, or pant quietly. I rarely use the last, but sometimes it's comforting when I'm not trying to sleep. The volume knob lets me quieten it down if necessary (have ot feel under the fur by the shoulder for the switch and the knob, as it's all mounted inside an old computer speaker that is secured inside a foamblock "ribcage" that itself is secured to the "spine"), which has the "bass" speaker inside it, and a "treble" speaker that is much smaller inside the head. The sound files were edited and EQ'd to distribute the sounds more realistically between those so they more or less sound like they should, as if they were coming from the correct parts of a dog's lungs and throat and nose. There's a USB extension cable glued into the board's port that I can pull out of the fur to reprogram the sounds for experiments, but I haven't done that in a while.

There's a whole lot here to digest, and I'll do my best to get to it all. I've been working 60+ hour weeks again for the past couple of months, so please don't take offense if I seem slow to respond. This is one of three links I try to check daily, so I'll get to it

If you think it's a lot to digest now, wait till I have eventually described everything I would like to happen, and how...

But most of the rest depends on how the in-development parts work out.

I'm in "no hurry" to get this all done; it happens as fast as it happens. Ideally, it'd all be really really fast, but I couldn't do it and wouldn't expect anyone else, especially volunteer help, to either.

I am hoping at some point things will just start coming together, like they usually do on other projects, but I don't think this is anywhere near that point--even the specifications/explanations haven't got there yet. :lol:

(I should probably really write a complete spec document...I started to, but as usual got distracted by actually doing things).

 

[amberwolf]

You mean it's not possible to monitor the current and use simple formulas to judge whether or not it's having difficulty, or at a midpoint or something?

Yes, and that has also been mentioned in some of the previous posts in the thread for ensuring it doesn't overload something, and can shutdown. But other things can cause current spikes, and the measurement of current doesn't tell you that something else is moving the actuated parts in some way that the actuators themselves are not (especially if they are not presently active).

So...that's one reason to use the IMU data to figure out where all the parts are and how they're moving and how fast and in what direction, etc., and keep track of that.

Also, (I think I posted this before, not sure; there is SO MUCH STUFF in my head about this project) : knowing the track of the motions, and such, it can be used to "teach" specific motions and behaviors, with some form of "direct motion capture" with this data as well. Faster than by-guess-and-by-gosh methods--let's say I wanted to teach it to lift a paw for a "high five": I could just grab the paw and move it and the rest of the limb in teh way and direction and speed I want the result to be. It could be somethign that requires a change of modes to do, so it's not "operational" during these times, but instead is just recording sensor data. Or it could be a "hey fido, lemme teach you something" command, etc. I have a number of possible ways it could work.

I have an entire system concept in my head for training motions this way, and editing them in a computer-based GUI before saving them to the robot's motion-memory-library, but I haven't posted that yet. I'll do that at some point, but right now it'd just get confusing with all this other stuff being discussed. By itself, it's pretty complicated...but would allow "anyone" to teach something specific to it in an easier way than teaching an actual dog to do it.

... I knew I should have taken at least an intro to electrical engineering course in school. I'm speechless.

What makes you speechless?


FWIW, I've never had any engineering courses (there's way too much math; I just couldn't follow it)...I learned some electrical and electronics stuff from a distant neighbor (who I only met because I crashed my bicycle down a gravel hill near their house) down the road in rural Texas as a kid, as he taught me about ham radio (I was KA5TWP but haven't done that stuff for decades); that was all vacuum-tube stuff though. Not long after, I had the short "technician" course at DeepFry...er...DeVry back in the late 80s, which taught me more modern electronics component-level troubleshooting and repair skills, but not how to design the stuff or much beyond the very very basic idea of how it works (just what was needed to understand why something might not be working).

So anything I happen to know i learned the hard way by building and breaking things, and having to figure out how to fix the things others didn't want so I could have one of whatever too-expensive-to-buy thing it was. Often enough, it's out of my skillset to deal with...and sometimes I can add enough to my skillset to do it, and sometimes I can't.

Same for mechanical stuff--no mech engineering; just mostly hacking existing structures/mechanisms/etc to misuse them in a way that does what I want...and reworking them as needed when my usage is outside the limitations of the existing stuff.


I know...well, more than enough to be less dangerous to myself and others than I was when I started out. :lol: And just enough to accomplish many of the things I set out to do in various projects...though they usually require multiple stages of refinements to end up doing what I really want them to. Sometimes I find out that what I *actually* wanted it to do is different from what I *thought* I wanted it to do, *after* I build it. Sometimes that's adaptable, and sometimes it's not.


Some things I just "do", like playing/creating music, or sketching/drawing, "kitbash" model building, sculpting, etc; just come to me, and "organically come together". But things like this project require considerably more actual work to achieve, and this is very difficult for me to do; I have a hard time pushing myself to do things that require actual sustained hard (mental) work (physically-exhausting hard work I do every day; I actually enjoy the doing much more than the designing...this is very hard to describe, and isn't accurate....so I'll stop).

 

[amberwolf]

Here's where I'm currently tripping over having more to say: I don't know of a good way to process a live, constant data stream like that. It's not something I've read much of any research about. Usually these things just sit there and do nothing until you come along and prompt them to do something by feeding a discrete chunk of data to them. A continuous stream of data is a little different, and I'm not having much luck coming up with a practical way of handling it.

A naive approach (which is often surprisingly effective, don't get me wrong) would be to just buffer the data for periods of time -- say 5 seconds at a time or something -- and then process that 5 second chunk before repeating the process over and over again. I wanted to have something more interesting to say than that, but it hasn't come to me yet.

Well... I guess, actually, you could do it with a standard LSTM (long short-term memory) model. It would just mean feeding timesteps to it piece by piece. Yeah, nevermind, I'm just being stupid -- that would work fine. Could even have some convolutions for good measure if you wanted to try them out.

I don't know enough about these things yet, so I don't undertstand why they would operate on a stream of data frames any differently than on sequentially-input single frames of the same data? I'm sure there's a reason....

Meaning...the data would be captured for one "frame" of movement (like a frame of video) for all the sensors. Then process that data to the "map". Then capture another frame, process it and alter the map (and store the previous map for later comparison, so changes over time can be looked at even if it's only a manual troubleshooting process, like playback of "WTF happened when we did *this*?" :lol: ).

(assuming we're still talking about data capture/processing of the IMU data for the map thing--if it's about something else, I missed it).

In my experience, designing an architecture to suit a problem is at least 50% trial and error. It's extremely difficult to know what would work best for a given application. It would be great if there were one model that was best for a big range of problems, but that's just not how it ends up working out in practice. You don't have to take my word for it, you can just go to paperswithcode.com and see the new architecture that's great for this-or-that on a daily basis. There's usually some credible sounding theory underlying them, but if you ask me, it really is just guessing and trying random stuff half the time to see what works best.

I'm very familar with this approach. :lol: In all sorts of things I do (even my "art") I usually have some idea of what to do, but not how to do it, so I end up doing this sort of thing....

Sometiems I have to design an approach to a troubleshooting problem. Meaning, there'sa problem I've never encoutnered, but it has elements I recognize, and so I have ot come up with a plan to test things to find the cause--but not knowing what the cause might be it's hard to do that, so I have to guess which things to test first without wasting too much time, or potentially making the problem worse, or creating new ones that will complicate the whole process. :/

 

[amberwolf]

I think our problem is primarily that all of your experience is in stuff that I have pretty much none in, and vice versa; definitely not your fault, I hope I didn't sound like I was implying that or anything.

Oh, don't worry--almost no one seems to really understand whatever it is I'm talking about. It is very hard for me to communicate the things in my head correctly, completely, but concisely, because there is so very very much going on in there all the time, all at the same time, and I have to pick out the bits that are related to the thing in question and sort them from all the other unrelated stuff, then condense it into words other people will understand***, and also not leave out stuff that is still in my head that didn't make it to the page yet...but that I already imagined writing...(I often have a very hard time distinguishing between the things I imagine and the things actually out there in front of me, because the imagining is extremely detailed and complete down to sensations...I can't even describe the problem well...****).

***this is particularly hard because few people care to "listen" to my "ramblings"; there is just too much information for them to process, but I have to include all of the details to get things across, or else they can't possibly understand what I mean, not really (I've tested this on various occasions), and generally they don't care enough about anything I have to say, not enough interest in whatever it is I'm interested in, to even attempt to absorb enough of the information for me to really communicate with them. I'm sure this has a teensy tiny bit to do with my exclusion from virtually all social circles.... It isn't that I'm any "smarter" than anyone else (I'm frequently quite a lot dumber despite what "IQ tests" say)...just that what I'm thinking is often so different from what they are that the common bits just arent' enough for reliable communication. (and I don't do realtime comms well because it takes me way too long to sort out all the stuff in my head to get the bits out I need to get across).

I try really hard to "be normal" whenever I'm just dealing with all the usual stuff between me and other people, but when it comes to things in my head to get them out to the world, especially when they're as complex as this project.....

**** I had added those stars to point to this explanation-continuation, but it got lost while I was writing the first (***) branch out. :/ So I am not certain how I intended to continue--all the bits and pieces of it are mixed up with all the other stuff again in my head.

Basically I need to "see" the raw data, or at least be able to imagine the 0s and 1s that would be fed into an artificial neural network, to have any more informed comments to add.

Well, I can post a stream of IMU data from a sensor in a given situation, if that would help. (have to figure out how to make it log that data, but I'm sure there's code out there for this already).

I'm not sure if that's what you're talking about or not. If you mean the whole data stream from the IMU network...that is likely to be at least a little while before I get that built, since I have to physically construct a skeleton to mount at least a dozen of them on first, mount them, wire them, then get the code working to read them all and write that data to some sort of organized file. If I didn't have to have the dayjob to survive, I could do the physical bits in a couple of weeks or so (depending on how fast the 3D printer can manufacture the skeleton parts). But the code stuff...I dont' even know enough to give an estimate on that part yet.

 

[riding_on]

I don't know enough about these things yet, so I don't undertstand why they would operate on a stream of data frames any differently than on sequentially-input single frames of the same data? I'm sure there's a reason....

Meaning...the data would be captured for one "frame" of movement (like a frame of video) for all the sensors. Then process that data to the "map". Then capture another frame, process it and alter the map (and store the previous map for later comparison, so changes over time can be looked at even if it's only a manual troubleshooting process, like playback of "WTF happened when we did *this*?" :lol: ).

(assuming we're still talking about data capture/processing of the IMU data for the map thing--if it's about something else, I missed it).

Look at it this way: how many updates per second is the hardware going to be sending out? Now compare that with my glazing over the process of using convolutions on a picture (where the matrix gets "rolled" over the pixel data -- which is a bunch of matrixes all stacked together itself -- one by one and "multiplied" by the matrix of the pixels in that square region of the image [multiplying the two matrices is my analogy for what a convolution really is doing]). It's not operating on 30 video frames per second, it's operating on one single bitmap of size 3x256x256 (3 channels [red-green-blue usually], each of which is a 256x256 grid/matrix/"bitmap" where the position inside those 256x256 (height x width or width x height) matrices indicate pixel position inside the image). If you want to use convolutions on a stream of video data, like a 30fps .avi file or something, usually people just increase the convolution operation to include an extra dimension (the time dimension). That's one of the reasons they're so convenient, is that they can operate on multiple dimensions (channels are just a dimension, abstractly) simultaneously. But then you're still just dealing with one, discrete chunk of data (the entire .avi file with [30 * DURATION_IN_SECONDS] x 3 x 256 x 256 values, or a 4 dimensional array/vector if viewed as 2D pixel maps with an added color axis/dimension, then with an added time axis/dimension).

So what I'm saying is that it isn't immediately obvious to me what a good way of turning a constant stream of input data into discrete chunks for processing is, in these circumstances. At some point there has to be an arbitrary buffer collection value, where processing is delayed while input values accrue and are collected into a chunk for processing. Everything kind of depends on how that's done, and how long of a delay is used. With something like time series forecasting (the project I mentioned previously about forecasting the available supply of water), you can choose to use hourly, daily, or even minute chunks of time. But it would take experimentation and a more concrete design (or real-world examples of the input data) for me to know where to begin, is pretty much what I'm getting at.

Remember, the entire duration that input is buffering and being collected into a chunk is time that there's literally nothing happening for. So even 5 seconds at a time would have a dramatic influence on reactivity.

 

[amberwolf]

Oh...ok. I didn't get that it was only working on totally static data.

I'll have to think about this because there is probably some solution, once I know more about how stuff like this works (but I have a long way to go on that).

As far as how much data there will actually be, I don't know. Minimum of 3 axes x 2 sensor types for each IMU, x however many sensors there are (best guess is a couple of dozen, minimum of a bit more than half that, potentially a few dozen if they're not as sensitive as I think they are). Don't know how often they have to be sampled to give reliable position and vector and velocity data--they can stream pretty quickly, from the demo code sketches I've played with so far.

The same data stream also includes the vibration/touch data embedded within the same channels, so a separate computing process would be used to find non-noise signals in it and then localize those to the specific sensor groups they came from, compare signal strengths with that and the map data generated by the first computing process above, to know where "exactly" the signal came from, so that plus the type of signal, over time, can be used to figure out what the interaction causing the signal actually is.

Maybe each of these things would all be separate computing processes running in delayed-parallel (staggered so the necessary data from one is available for the next), possibly on separate MCUs with separate code?

ATM I'm still designing "in a vacuum" since I don't yet know enough about the physical hardware capabilities or the software capabilities...I only know what I want to happen, but still have to learn much more to *make* that happen.

 

[riding_on]

Oh...ok. I didn't get that it was only working on totally static data.

I'll have to think about this because there is probably some solution, once I know more about how stuff like this works (but I have a long way to go on that).

Just because I personally can't think of a good way of doing it without sitting down to actually do it, doesn't mean that it can't be done or that it's too hard to do. I'm not a super genius or anything -- just your regular next door kind, hah. I hope I didn't discourage you

 

[amberwolf]

Well, as you said, you'll need data to work with, so other than posting ideas as they come to me and I work them out, I'm trying to focus on building sufficient hardware to get some of that data. Since I will need a skeleton anyway, I'm figuring out how to 3D print the pieces so they'll go together and stay that way without breaking, and pass the cabling along them in some funcitonal way. The cabling will at least for now just be bicycle brake/shifter cables/housings, since I have those, pulled by the servos I have.

But all I need for the data collection is the functional skeleton, to be moved by hand, with the IMUs mounted to the bones, and wired up to one or more MCUs to collect and store the raw data.

Once I have that I can stick that in a file along with the motions used to create the data, the data format, code that generated it, and the hardware design, here in the thread, and then you can see if it is usable to interpret.

Does that sound like a useful plan?


FWIW, other than the (lots of) very specific but miscellaneous bits of knowledge I've collected and blended together improperly in my head, I'm no genius either--at best average, and more often retarded about many things, with wacky ideas that almost never work out (certainly not as planned).

 

[amberwolf]

Still trying to learn enough to program the IMUs to do things; found this

Motion Sensing GloveGesture Control Smart Car for Arduino — keyestudio WiKi documentation

that may be useful.

Haven't yet been able to finish editing the skeleton model(s) enough to print them and build as a testbed to get the hardware installed.

Then I have to learn how to create code that will capture and log the data from multiple sensors at once, into a file (some kind of database; don't know what or how yet). Not very good at learning this stuff, so will probably be a long while at the present rate.

 

[amberwolf]

Tried to go back to learning some code by using some other project examples, but am again stonewalled. As usual, I can't even compile the project and have found no workarounds. These are the kinds of things that deter me from coding, because I can't understand why it doesn't work, and there's no options in the software to either tell me what's wrong, or to let me manually fix them in ways suggested by online posts for similar (or even identical) problems.


The present example is fairly unrelated to this thread's project, but have a cheap ($3) NodeMCU with a tiny OLED built in

https://www.aliexpress.us/item/3256806050972938.html

and before I tried making it detect and display the angles from one of these MPU sensors (to see if I could learn to code something for that from scratch instead of just copying an example code set), I thought I'd try to get one of the various very simple (haha) ESP32 Oscilloscope projects (the first one of those listed below; there's several others out there too) adapted to it just to see if they would run on it at all, and show me something interesting on the display...and it might actually be useful, too.

DIY ESP32 Oscilloscope

Here we build an ESP32-based Oscilloscope with a 1.69" TFT Display that is affordable, portable, and simple to construct.

circuitdigest.com

Arduino Based Dual Channel Digital Oscilloscope

Arduino Based Dual Channel Digital Oscilloscope: It is like a mini atom bomb, In a piece of single code you will get a lot of features and I fully respect the effort of the owner.

www.instructables.com

But...the project requires several "includes" such as adc.h, which in turn has it's own includes like gpio.h... etc.

I managed to force the system to find the adc.h (which no version of is anywhere on my computer) by locating a version of it in an ESP32 library here

GitHub - husarnet/esp-idf: Espressif ESP-IDF with added support for Husarnet

Espressif ESP-IDF with added support for Husarnet. Contribute to husarnet/esp-idf development by creating an account on GitHub.

github.com

and just putting the file inside my "libraries" folder in a subfolder "ADC", and then using the ENTIRE LOCAL PATH to that file, since no shorter version would work, and the Arduino IDE is too stupid to bother checking for files anywhere you don't specifically completely locate for it.

But the technique didn't work for the sub-includes, even if I edit the adc.h file for the paths to them, even if I stick them in the same ADC folder and edit the adc.h file itself to point to those exact full paths.

I opened the adc.h file in the IDE and tried compiling just that, but even with those edits it won't compile, sticking on being unable to find gpio.h, even though I stuck that in there the same way I did the adc.h, and put the path in, etc. I wasted about four hours on internet searches and attempts to either use the fixes I found, adapt them to my system, or just try logical derivatives of them or of the things I'd done for the adc.h that worked. No luck.


Aside from my lack of ability to really grasp what functions can be done by what kinds of software commands (meaning, while I understand the idea behind them, I don't know all of the commands and how to build things with them...no, that's not really what I mean; I don't even know how to say what I mean). I don't know how to even conceptualize how to turn an idea I have about what needs to be done into actual code (not knowing what code does what, and finding that I don't understand what other people mean when they discuss what can be done to do these things), stuff like the above is what keeps me from doing anything about the software for this project, and sends me on some pretty deep depressions about it that keep me from coming back to it to even try things very often.


I usually end up closing all the coding stuff and opening SONAR and making music, because at least that is usually calming, if often difficult; I understand what's going on in there. I spent the last couple of months more on The Moon, It Read To Me, And It Was Bright than coding attempts, because as hard (and often frustrating) as it is to do the music, it's MUCH easier than the coding, and at least it provides some reward in that I can get results I can experience...all I get out of coding attempts are frustration and stress.

(I also always have...not sure what to call them. Brain-shifts; I have so many different things in my head that they don't all fit at once. Sometimes the one I am trying to work on has taken a trip to an alternate universe, and I can't access all the required info for it, can't get it all "back" in my head, so I have to work on something else until it comes back. (sometimes stress from daily events builds up and drives *all* of it away for a while.) That delays projects as much as any of the other things I've discussed here. :/ )


I understand what *can* be done...I know the things I generally need to do to do them, but I have no idea what actual code to use to do them. And I don't understand how to figure that out from scratch.

And even when I find existing code that seems to be able to do something I want to do, I usually can't even get it to compile to be able to run it and see what happens. (sometimes I get lucky and it just works, or I can hack at it with logical deductions and make it work on my hardware....but almost never does this happen).

I can't afford to pay a good dedicated programmer to do it, but that's the only option I can think of to get anywhere; I've been trying to learn actual coding for so long (and have actualy done coding for simple stuff many years ago, in BASIC and even assembly, so I understand the principles), with just frustration at every attempt to do even the simplest things with existing tutorials (most of the time even simple things like Hello World code examples don't even work for me, just copied and pasted in, and I can't see why not).

Because I need this project to happen, I won't give up on it...but I am stuck; I'll keep developing the ideas themselves, I'll keep working on the hardware (3d-printed "skeleton" models and then actual parts, motor drives, molds for parts that have to be cast, patterns to make the exterior coverings from, etc), and I'll keep poking at coding in the slim hope I'll get that "eureka" moment.


So...for the moment, at least, that leaves this project fairly dead-ended at this point.

 

[AnthonyC]

I came across these projects linked below a while back, there is something special about wolves and they've inspired some very dedicated work.

If you're not yet steeped in Arduino what about moving to a higher level with python instead? The devt environment needs less setting up than for Arduino and python's expressiveness is better for data crunching and complexity.

Your project outline is very ambitious in the amount of detail to complete, even if it were all straightforward. Keeping with the high level theme, have you started at your own neural network to understand more about what it is you're getting from the wolfies? Maybe whatever you need most, responsiveness say, could be tackled early on in some direct way (such as the sounds adapting with handling), with your full vision following on after. (Sorry if I've missed this up-thread.)

WolfTronix Animatronic wolf head

WolfTronix Animatronic wolf head

TheMariday.com - Highbeam

TheMariday.com - Highbeam

Highbeam The Idea lorem ipsum The People lorem ipsum The Hardware lorem ipsum The Software lorem ipsum

www.themariday.com

multiwingspan

multiwingspan

MicroPython - Python for microcontrollers

MicroPython - Python for microcontrollers

MicroPython is a lean and efficient implementation of the Python 3 programming language that includes a small subset of the Python standard library and is optimised to run on microcontrollers and in constrained environments.

micropython.org

How To Use A Servo With Raspberry Pi Pico - YouTube

(Edit: moved link to micropython section)

 

[nicobie]

So...for the moment, at least, that leaves this project fairly dead-ended at this point.

Hey, just wait until you get older...

It gets worse.