Cycle Analyst Serial Data Truncation

Get all your technical information about electric bikes here.
Post Reply
yandemontreal   1 mW

1 mW
Posts: 11
Joined: Mar 18 2018 3:49pm
Contact:

Cycle Analyst Serial Data Truncation

Post by yandemontreal » May 07 2018 8:32am

Hi all,

I've setup a Pi Zero to grab data from the CA serial port. For now, I simply run Raspbian and cat the serial device (with the Grin FTDI cable) onto a file. It runs generally well. I also grab GPS data directly from the UART pins in the same manner.

I have this issue with the CA log where approximately one in every 13 to 15 lines gets truncated. That's at a 1Hz data rate. At 10Hz, the loss was more frequent and not as constant.

I suspect it's my logging method that's wrong. Has anyone else experienced this?

User avatar
teklektik   10 GW

10 GW
Posts: 4020
Joined: Mar 26 2011 1:15pm
Location: CT, USA

Re: Cycle Analyst Serial Data Truncation

Post by teklektik » May 07 2018 10:51am

yandemontreal wrote:
May 07 2018 8:32am
I have this issue with the CA log where approximately one in every 13 to 15 lines gets truncated. That's at a 1Hz data rate.

I suspect it's my logging method that's wrong. Has anyone else experienced this?
I believe the difficulty is with your input mechanism.

There is a little weirdness in the CA serial I/O that has always been there and is fixed in the upcoming 3.11 release, but there have been no reports of data loss in spite of that. I regularly log thousands of lines at 10hz with no issue using either puTTYCyg or a direct cat of the tty device under Cygwin.

In the 3.1 release, the CA detects incomplete record transmissions and appends an exclamation point to truncated lines before transmitting the new record. This is done as a cue to postprocessing software that the data is explicitly incomplete and not simply 0. If you are not seeing these on your truncated lines then there is a good chance the data actually went out but you missed it somehow.

This sort of thing can occur if framing or overrun errors occur as the receiver coughs a bit trying to assemble records into canonical form. You might verify your port config. See quickie test run below.

That said, you may have discovered a real issue. I suggest that you start by using a terminal emulator on your PC or phone to capture the data stream using a different proven platform. This is described in the Guide in section "6.15 Logging Cycle Analyst Data with Phone or PC". If you see the same lossy behavior then you may be on to something.

SerialOutCygwin.png
CAv3.1 @ 10Hz data rate, Cygwin, Win7, Dell Latitude D630
SerialOutCygwin.png (110.47 KiB) Viewed 415 times
Visit Grin Technologies at www.ebikes.ca
Build Thread: 2WD Yuba Mundo V4

yandemontreal   1 mW

1 mW
Posts: 11
Joined: Mar 18 2018 3:49pm
Contact:

Re: Cycle Analyst Serial Data Truncation

Post by yandemontreal » May 07 2018 8:11pm

teklektik wrote:
May 07 2018 10:51am
I believe the difficulty is with your input mechanism.
I'll have to agree. I did cat the line (to stdout though) without issue when connected directly to a PC. I've tried it both with PuTTY and through Cygwin.

I compared the line settings from Raspbian with what you posted and they are quite different. I think the answer must lie therein.

During my first attempts, I would get extra linefeeds, and after some googling I ended up setting the line 'raw' with

Code: Select all

stty -F /dev/ttyUSB0 raw
and that took care of it.

The line settings after this became:

Code: Select all

speed 9600 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>; 
start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; discard = ^O; min = 1; time = 0;
-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany -imaxbel -iutf8
-opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke -flusho 
-extproc
After seeing your settings, I tried to copy them as much as possible (I don't/haven't tried to understand most of these settings):

Code: Select all

speed 9600 baud; rows 0; columns 0; line = 0;
intr = <undef>; quit = <undef>; erase = <undef>; kill = <undef>; eof = <undef>; eol = <undef>; 
eol2 = <undef>; swtch = <undef>; start = ^Q;
stop = ^S; susp = <undef>; rprnt = <undef>; werase = <undef>; lnext = <undef>; discard = ^O; min = 0;
 time = 0; -parenb -parodd -cmspar cs8 -hupcl -cstopb -cread clocal -crtscts
-ignbrk -brkint ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany -imaxbel -iutf8
-opost -olcuc -ocrnl -onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon -iexten -echo -echoe -echok -echonl -noflsh -xcase -tostop -echoprt -echoctl -echoke 
-flusho -extproc
But that ended up being worse: I only got part of one line, and then nothing. The other time it was one line of garbage (probably because I plugged in during an output?).

I'll be trying other combinations when I have time.

yandemontreal   1 mW

1 mW
Posts: 11
Joined: Mar 18 2018 3:49pm
Contact:

Re: Cycle Analyst Serial Data Truncation

Post by yandemontreal » May 08 2018 7:39am

Here's some more interesting info.

From the default line settings, I only -inlcr to get rid of the extra linefeed, what I should have done from the beginning.

I'm at 10Hz and as expected I get a lot more truncated lines. Now I don't see the exclamation point you mentioned. But curiously, I logged for 5 seconds and got a total of 100 lines, while I was expecting 50. Sure enough, if I discard all the truncated lines, I'm left with exactly 50 lines. At least it reassures me that I'm not missing any data.

The default line settings were:

Code: Select all

speed 9600 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z
min = 1; time = 0;
-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff -iuclc -ixany -imaxbel -iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke -flusho -extproc

User avatar
teklektik   10 GW

10 GW
Posts: 4020
Joined: Mar 26 2011 1:15pm
Location: CT, USA

Re: Cycle Analyst Serial Data Truncation

Post by teklektik » May 08 2018 10:19am

Not sure what's going on with that stty implementation. The example above ran fine with just resetting the baudrate.

For a shotgun fix, you might try:

Code: Select all

stty -F /dev/ttyS7 sane
stty -F /dev/ttyS7 speed 9600 -parenb -parodd cs8
Otherwise, I would think it's largely a matter of disabling all the interactive and eol features (except to discard CRs) and all modem signal and flow control, but you seem to be heading down that path already...

You might also try loading up puTTYCyg and configuring the port that way - which should work fine - then try running a cat on the preconfigured port which should then work okay, too. Finally, use stty to pull the working config. Just a thought....

Anyhow, this is doable - you just seem to have a stubborn stty....
Visit Grin Technologies at www.ebikes.ca
Build Thread: 2WD Yuba Mundo V4

yandemontreal   1 mW

1 mW
Posts: 11
Joined: Mar 18 2018 3:49pm
Contact:

Re: Cycle Analyst Serial Data Truncation

Post by yandemontreal » May 09 2018 5:43pm

Alright, I finally got it working with just
-inlcr -echo
Thanks for helping me persist! :D

User avatar
teklektik   10 GW

10 GW
Posts: 4020
Joined: Mar 26 2011 1:15pm
Location: CT, USA

Re: Cycle Analyst Serial Data Truncation

Post by teklektik » May 09 2018 7:10pm

Excellent!
I'm a little amazed that the tty driver defaults to 'echo', but whatever.... :D
Thanks for posting back with results - I'm going to link this to the 'Posts of Interest' in the CA Guide download post.
Visit Grin Technologies at www.ebikes.ca
Build Thread: 2WD Yuba Mundo V4

yandemontreal   1 mW

1 mW
Posts: 11
Joined: Mar 18 2018 3:49pm
Contact:

Re: Cycle Analyst Serial Data Truncation

Post by yandemontreal » May 10 2018 1:45pm

Pretty amazing indeed. And for reference, this is the Raspbian Stretch Lite distribution.

User avatar
rberger   10 W

10 W
Posts: 76
Joined: Mar 09 2015 2:10am
Location: Saratoga, CA

Re: Cycle Analyst Serial Data Truncation

Post by rberger » May 12 2018 2:38am

Got some ruby code I run on Raspberry Pi for the Cycle Analyst 3, Phaserunner and GPS that are all connected to the Raspberry PI via USB (using the Grintech USB cables for CA and Phaserunner):

cycle_analyst_logger It can log all three. It outputs a CSV file with data from the CA/Phaserunner/GPS plus a separate NMEA GPS. It has a utility to take the csv file suitable for http://www.ebikes.ca/tools/trip-analyzer.html

It uses phaserunner library for accessing the phase runner stuff

Besides the source code in GitHub (Links above), you can just download and install the cycle_analyst_logger Ruby Gem assuming you have a relatively modern ruby environment set up.

Still a bit rough around the edges but is generally working for me. Works with macOS as well as Linux too.
2011 ICE Adventure Trike Converted to 1500W DD Hub Pedalec in 2017

Post Reply