[STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Discussions about the STM32generic core
User avatar
Pito
Posts: 1529
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by Pito » Mon Aug 21, 2017 9:43 am

The Serial.something comes historically from RS232-like serial UART communication, where the handshaking with the Host is managed by RTS/CTS, DTR/DSR etc., or via XON/XOFF, or done at application level via XYZMODEM/KERMIT etc.
Without such handshaking it could come to data loss with RX/TX. But that handshaking is usually not used with duinos. So Serial.something can loose data when internal ringbuffers full or slow reading them, or something like that.

SerialUSB uses a sophisticated USB protocol, which inherently includes handshaking. So the handshaking at "packets" level is there. It needs to be used and then you cannot "loose" any information while RX/TX.
The people may discuss what should happen when we use USB layer for "Serial" emulation - whether we shall propagate RTS/CTS or that kind of signals, etc. What I've seen in various discussions on this topic - even they say USB CDC includes this kind of flags (OUT command 0x22 (SET_CONTROL_LINE_STATE) RTS/DTR bits in CDC_ACM) - it is not necessary with USB, as you can tell the other side to "stop sending I cannot read the new packet" by NAK'ing, the other side will repeat again till ACKed.

FYI: Similar discussion at PaulS forum - an .ino sketch and Python script for testing and monitoring the CDC serial..
https://forum.pjrc.com/threads/33167-US ... -detection
maybe we can reuse that somehow.

About NAK'ing IN/OUT USB packets:
http://nuttx.org/doku.php?id=wiki:nxint ... sb-out-nak
Pukao Hats Cleaning Services Ltd.

User avatar
Pito
Posts: 1529
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by Pito » Mon Aug 21, 2017 6:24 pm

FYI - this is how the "USBlyzer" (not free, 33d eval) shows the transfer between MMini and TeraTerm (Libmaple F1).
The USBlyzer provides huge amount of info, so I had to cut off a small chunk only - which fits as the attachment.
You may see the payload UUUU data are fragmented into 1-16/414-426bytes large chunks. No Idea what "4096 buffer" means.
The transactions are marked successful.
USBlyzer 3.JPG
USBlyzer 3.JPG (136.28 KiB) Viewed 231 times
Pukao Hats Cleaning Services Ltd.

User avatar
Rick Kimball
Posts: 1014
Joined: Tue Apr 28, 2015 1:26 am
Location: Eastern NC, US
Contact:

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by Rick Kimball » Mon Aug 21, 2017 7:15 pm

@Pito have you tried it with "putty" instead of TeraTerm?
-rick

victor_pv
Posts: 1650
Joined: Mon Apr 27, 2015 12:12 pm

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by victor_pv » Tue Aug 22, 2017 5:19 am

Pito the Windows driver is definitely doing some buffering on it's end, since the bulk transfer is only for 64bytes, hard coded in the libmaple driver.
I wonder is the 4096 is some information the host sends to the board to indicate how much buffer space it currently has.

User avatar
Pito
Posts: 1529
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by Pito » Tue Aug 22, 2017 9:08 am

@Rick: I tried with HyperTerminal (the same result - see above) and with minicom (ubuntu in virtualbox). With minicom I did not make any captures.

@Victor: the "4096b buffer" could be some misinterpretation of data by the USBlyzer, or handshake..
UPDATE: it seems the usbser.sys requests 4kB bulk IN..

The Seq. numbers always refer packets like 128-127 where the 127 was the "4096b buffer" packet so it could be the 128th packet responded to the 127th..

What is interesting is the result - the payload chunks sizes - are similar to what I got from Wireshark.
It sends in ~1ms (the packet period) ~16 or ~420bytes ("random size"). That is something which needs to be understood.

While doing Serial.write('U') in a loop you fill in the buffers fast, so I would expect the outgoing payload packets will always contain the amount of UUUs equal to the lowest layer buffer size.

UPDATE: MMini Libmaple against Putty (Win7_64b) - the same results as the above with Wireshark and USblyzer.
Putty.JPG
Putty.JPG (226.03 KiB) Viewed 193 times
.
FYI - this is how a CDC transfer looks like on an atxmega @32MHz sending 1MB/sec to PC (from http://www.avrfreaks.net/forum/bulk-tra ... te-asf-usb):
ATXmega.JPG
ATXmega.JPG (90.43 KiB) Viewed 180 times
.
That somehow supports my hypothesis above on the amount of UUUs in the payload packet.
I firmly believe soon or later we will achieve the 1MBytes/sec (with no data loss).. :)
Pukao Hats Cleaning Services Ltd.

User avatar
Pito
Posts: 1529
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by Pito » Tue Aug 22, 2017 1:19 pm

This is with MMini, Libmaple F1, with this vitor_boss' mod of my sketch (Serial.write(buf..):

Code: Select all

void loop() {
  uint32_t i;
  uint8_t x = 85;
  uint32_t elapsed = micros();
  uint8_t buf[bufsize];

  for (i = 0; i < bufsize; i++) { buf[i] = x; }

    elapsed = micros();
    for (i = 0; i < TXCHARS; i+=bufsize) {
      Serial.write(buf, bufsize);
    }
    elapsed = micros() - elapsed;
For bufsize >= 4 (four) you get always this nice capture (against TT, speed 420-440kB/sec):
.
Libmaple F1 Serial(buf_bigger than 4).JPG
Libmaple F1 Serial(buf_bigger than 4).JPG (161.06 KiB) Viewed 174 times
.

STM32Generic
This as above but with STM32Generic, CDC_BUF 128, bufsize 4 (..32), speed 64kB/sec:
.
STM32Generic CDC_BUFF 128.JPG
STM32Generic CDC_BUFF 128.JPG (166.63 KiB) Viewed 166 times
.
STM32Generic, CDC_BUF 128, bufsize = 64, speed 420kB/sec:
STM32Generic CDC_BUFF 128 bufsize eq or big than 64.JPG
STM32Generic CDC_BUFF 128 bufsize eq or big than 64.JPG (134.16 KiB) Viewed 165 times
Last edited by Pito on Tue Aug 22, 2017 2:07 pm, edited 2 times in total.
Pukao Hats Cleaning Services Ltd.

User avatar
Pito
Posts: 1529
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by Pito » Tue Aug 22, 2017 1:52 pm

Problem: STM32Generic with CDC_BUF 128, and bufsize = 256 (speed ???)
3seconds between packets
STM32Generic CDC_BUFF 128 bufsize 256.JPG
STM32Generic CDC_BUFF 128 bufsize 256.JPG (134.05 KiB) Viewed 156 times
And STM32Generic with CDC_BUF 4096, and bufsize = 64 (speed 420kB/sec)
STM32Generic CDC_BUFF 4096 and bufsize 64.JPG
STM32Generic CDC_BUFF 4096 and bufsize 64.JPG (124.48 KiB) Viewed 160 times
STM32Generic with CDC_BUF 4096, and bufsize = 1024 (speed 420kB/sec)
STM32Generic CDC_BUFF 4096 and bufsize 1024.JPG
STM32Generic CDC_BUFF 4096 and bufsize 1024.JPG (160.48 KiB) Viewed 159 times
.
.
Note: I have not checked the number of bytes received.
Pukao Hats Cleaning Services Ltd.

victor_pv
Posts: 1650
Joined: Mon Apr 27, 2015 12:12 pm

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by victor_pv » Tue Aug 22, 2017 2:28 pm

Why did you mark this as problem?
Problem: STM32Generic with CDC_BUF 128, and bufsize = 256 (speed ???)
3seconds between packets

User avatar
Pito
Posts: 1529
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by Pito » Tue Aug 22, 2017 4:48 pm

Because of 3secs between the packets when the bufsize in the sketch is 256bytes (or 512, 1024 etc)..
That is even visible in TT as the 4kB chunks always nap for 3secs..

Another test - counting bytes - logged into TT file.
To make it simple I've TXed 1.280.000 bytes

Code: Select all

#define bufsize 64
#define TXCHARS bufsize*20000
Libmaple F1, bufsize 64 in sketch, logged 1.280.000 bytes -> OK.
The final packets:
Libmaple F1 finish.JPG
Libmaple F1 finish.JPG (47.27 KiB) Viewed 141 times
.
STM32Generic, bufsize 64 in sketch,, CDC BUF 128, logged 1.277.952 bytes -> 2048 less, not OK.
The final packets (you may see we miss the last 2048b packet):
STM32Generic finish.JPG
STM32Generic finish.JPG (40.73 KiB) Viewed 141 times
.
.
Note: in the USBlyzer the left hand side number called Seq (I wrongly named "packet"):
..Each item in the Capture List corresponds to a request that has passed through the particular device and has its own unique Sequence number..
..You will see two sequence numbers in the rows corresponding to the IRP's completion. The second sequence number corresponds to the point in the list where the IRP (I/O Request Packets) was issued..
So the Seq could be several packets long (ie. see 2-6ms long Seqs above).
Pukao Hats Cleaning Services Ltd.

danieleff
Posts: 336
Joined: Thu Sep 01, 2016 8:52 pm
Location: Hungary
Contact:

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by danieleff » Wed Aug 23, 2017 4:39 pm

Pito wrote:
Tue Aug 22, 2017 4:48 pm
Libmaple F1, bufsize 64 in sketch, logged 1.280.000 bytes -> OK.
My problem is that I do get losses with libmaple maple mini TeraTerm (4.95 (SVN# 6761) May 31 2017 20:25:51) (Win10, usbser.sys 10.0.14393.0). Data received in log file from 1272912 to 1279552.

Edit: If I run other CPU intensive program simultaneously (https://www.mersenne.org/download/), the problem goes away, and I get all the data from the above test.

Post Reply