Cycle Analyst Serial Data Truncation

yandemontreal · May 7, 2018

Hi all,

I've setup a Pi Zero to grab data from the CA serial port. For now, I simply run Raspbian and cat the serial device (with the Grin FTDI cable) onto a file. It runs generally well. I also grab GPS data directly from the UART pins in the same manner.

I have this issue with the CA log where approximately one in every 13 to 15 lines gets truncated. That's at a 1Hz data rate. At 10Hz, the loss was more frequent and not as constant.

I suspect it's my logging method that's wrong. Has anyone else experienced this?

teklektik · May 7, 2018

yandemontreal said:
I have this issue with the CA log where approximately one in every 13 to 15 lines gets truncated. That's at a 1Hz data rate.

I suspect it's my logging method that's wrong. Has anyone else experienced this?

I believe the difficulty is with your input mechanism.

There is a little weirdness in the CA serial I/O that has always been there and is fixed in the upcoming 3.11 release, but there have been no reports of data loss in spite of that. I regularly log thousands of lines at 10hz with no issue using either puTTYCyg or a direct cat of the tty device under Cygwin.

In the 3.1 release, the CA detects incomplete record transmissions and appends an exclamation point to truncated lines before transmitting the new record. This is done as a cue to postprocessing software that the data is explicitly incomplete and not simply 0. If you are not seeing these on your truncated lines then there is a good chance the data actually went out but you missed it somehow.

This sort of thing can occur if framing or overrun errors occur as the receiver coughs a bit trying to assemble records into canonical form. You might verify your port config. See quickie test run below.

That said, you may have discovered a real issue. I suggest that you start by using a terminal emulator on your PC or phone to capture the data stream using a different proven platform. This is described in the Guide in section "6.15 Logging Cycle Analyst Data with Phone or PC". If you see the same lossy behavior then you may be on to something.

yandemontreal · May 7, 2018

teklektik said:
I believe the difficulty is with your input mechanism.

I'll have to agree. I did cat the line (to stdout though) without issue when connected directly to a PC. I've tried it both with PuTTY and through Cygwin.

I compared the line settings from Raspbian with what you posted and they are quite different. I think the answer must lie therein.

During my first attempts, I would get extra linefeeds, and after some googling I ended up setting the line 'raw' with

Code:

stty -F /dev/ttyUSB0 raw

and that took care of it.

The line settings after this became:

Code:

speed 9600 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>; 
start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; discard = ^O; min = 1; time = 0;
-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany -imaxbel -iutf8
-opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke -flusho 
-extproc

After seeing your settings, I tried to copy them as much as possible (I don't/haven't tried to understand most of these settings):

Code:

speed 9600 baud; rows 0; columns 0; line = 0;
intr = <undef>; quit = <undef>; erase = <undef>; kill = <undef>; eof = <undef>; eol = <undef>; 
eol2 = <undef>; swtch = <undef>; start = ^Q;
stop = ^S; susp = <undef>; rprnt = <undef>; werase = <undef>; lnext = <undef>; discard = ^O; min = 0;
 time = 0; -parenb -parodd -cmspar cs8 -hupcl -cstopb -cread clocal -crtscts
-ignbrk -brkint ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany -imaxbel -iutf8
-opost -olcuc -ocrnl -onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon -iexten -echo -echoe -echok -echonl -noflsh -xcase -tostop -echoprt -echoctl -echoke 
-flusho -extproc

But that ended up being worse: I only got part of one line, and then nothing. The other time it was one line of garbage (probably because I plugged in during an output?).

I'll be trying other combinations when I have time.

yandemontreal · May 8, 2018

Here's some more interesting info.

From the default line settings, I only -inlcr to get rid of the extra linefeed, what I should have done from the beginning.

I'm at 10Hz and as expected I get a lot more truncated lines. Now I don't see the exclamation point you mentioned. But curiously, I logged for 5 seconds and got a total of 100 lines, while I was expecting 50. Sure enough, if I discard all the truncated lines, I'm left with exactly 50 lines. At least it reassures me that I'm not missing any data.

The default line settings were:

Code:

speed 9600 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z
min = 1; time = 0;
-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff -iuclc -ixany -imaxbel -iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke -flusho -extproc

teklektik · May 8, 2018

Not sure what's going on with that stty implementation. The example above ran fine with just resetting the baudrate.

For a shotgun fix, you might try:

Code:

stty -F /dev/ttyS7 sane
stty -F /dev/ttyS7 speed 9600 -parenb -parodd cs8

Otherwise, I would think it's largely a matter of disabling all the interactive and eol features (except to discard CRs) and all modem signal and flow control, but you seem to be heading down that path already...

You might also try loading up puTTYCyg and configuring the port that way - which should work fine - then try running a cat on the preconfigured port which should then work okay, too. Finally, use stty to pull the working config. Just a thought....

Anyhow, this is doable - you just seem to have a stubborn stty....

yandemontreal · May 9, 2018

Alright, I finally got it working with just [pre]-inlcr -echo[/pre]

Thanks for helping me persist!

teklektik · May 9, 2018

Excellent!
I'm a little amazed that the tty driver defaults to 'echo', but whatever....

Thanks for posting back with results - I'm going to link this to the 'Posts of Interest' in the CA Guide download post.

yandemontreal · May 10, 2018

Pretty amazing indeed. And for reference, this is the Raspbian Stretch Lite distribution.

rberger · May 12, 2018

Got some ruby code I run on Raspberry Pi for the Cycle Analyst 3, Phaserunner and GPS that are all connected to the Raspberry PI via USB (using the Grintech USB cables for CA and Phaserunner):

cycle_analyst_logger It can log all three. It outputs a CSV file with data from the CA/Phaserunner/GPS plus a separate NMEA GPS. It has a utility to take the csv file suitable for http://www.ebikes.ca/tools/trip-analyzer.html

It uses phaserunner library for accessing the phase runner stuff

Besides the source code in GitHub (Links above), you can just download and install the cycle_analyst_logger Ruby Gem assuming you have a relatively modern ruby environment set up.

Still a bit rough around the edges but is generally working for me. Works with macOS as well as Linux too.

Cycle Analyst Serial Data Truncation

yandemontreal

1 mW

teklektik

100 MW

yandemontreal

1 mW

yandemontreal

1 mW

teklektik

100 MW

yandemontreal

1 mW

teklektik

100 MW

yandemontreal

1 mW

rberger

10 W

Similar threads