[STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Discussions about the STM32generic core
User avatar
Pito
Posts: 1531
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by Pito » Sat Aug 19, 2017 8:05 am

Libmaple F1 - similar to 32Generic but much more UUs inside a packet:
Libmaple F1 capture.JPG
Libmaple F1 capture.JPG (227.8 KiB) Viewed 185 times
Sometimes a malformed packet:
Libmaple F1 capture 2.JPG
Libmaple F1 capture 2.JPG (202.11 KiB) Viewed 184 times
The list of Malformed in the upload:
Libmaple F1 malformed.JPG
Libmaple F1 malformed.JPG (220.37 KiB) Viewed 183 times
.
Interestingly the last packet (N. 3673) is not malformed :)

My current understanding is we loose sync somehow, thus the 5555 mess. It could be seen above as the "normal handshake" (till packet N. 65) went via 'Unknown type" and 2x Malformed packets and then into the "5555" protocol mess.
Based on what I've seen with corsair flash stick, all packet's protocol names must be USB related (not 5555, nor Ethernet).
No idea whether it comes from stm32 or TeraTerm, our experts may dig into..

Note: all above wireshark experiments were done with TX=800k (instead of 1mil chars sent) as the default USBPcap buffer is 1MB (size could be changed but I have not played with the size so far..).
Pukao Hats Cleaning Services Ltd.

User avatar
Pito
Posts: 1531
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by Pito » Sat Aug 19, 2017 10:15 am

Libmaple F1 against HyperTerminal (227kB/s)
Libmaple F1 against HyperTerminal.JPG
Libmaple F1 against HyperTerminal.JPG (245.55 KiB) Viewed 164 times
.
What are the two sources/destinations 2.10.0 and 2.10.1 ??
It could be all that Ethernet errors are USB Packet capture limitations/features/bugs??

http://desowin.org/usbpcap/capture_limitations.html

Btw HyperTerminal has captured 800k chars into a file (ok).

UPDATE1: I think there could be a bug in the USBPcap - https://github.com/desowin/usbpcap/issues/42
so take the above info as the demonstration the USB traffic could be captured and analyzed :)
There are maybe other tools as well..

UPDATE2: Do not panic - while watching Sample USB captures at wireshark's wiki I see the Malformed packets are a standard occurrence..
Tried ST-Link, usb flash stick and 2 cameras with captures without any "visible" anomalies, all packet's protocol names starting with "USB..", no "Ethernets" or any weird "5555".. Nice clean something..
Last edited by Pito on Sat Aug 19, 2017 12:53 pm, edited 4 times in total.
Pukao Hats Cleaning Services Ltd.

zmemw16
Posts: 1422
Joined: Wed Jul 08, 2015 2:09 pm
Location: St Annes, Lancs,UK

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by zmemw16 » Sat Aug 19, 2017 12:04 pm

if last packet isn't full maybe try that ?
wondering if sending packets all 1 byte below max size would change something ?
srp

User avatar
Pito
Posts: 1531
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by Pito » Sat Aug 19, 2017 12:38 pm

FYI: Upload via DFU (maple_loader v0.1, host, from Sloeber) into MapleMini (dev 2.39.0) - the payload packets (1052bytes) with some handshaking (only a small chunk of data shown, no anomalies there till the end, all clean, only 2 Malformed packets out of about ~60 during enum/init phase):
MapleMini DFU BIN UPLOAD.JPG
MapleMini DFU BIN UPLOAD.JPG (240.49 KiB) Viewed 142 times
Pukao Hats Cleaning Services Ltd.

victor_pv
Posts: 1654
Joined: Mon Apr 27, 2015 12:12 pm

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by victor_pv » Sat Aug 19, 2017 2:59 pm

Guys, with all the cores currently there is a TX timeout, if the host is not pulling packets as fast as the MCU can send them, there will be loses since the timeout will expire and the TX function will return before it sent everything.

That's one of the reasons I modified Pito test sketch to send blocks rather than 1 byte write, so I was doing bigger transactions rather than many very small ones.

I think for testing we could comment out the timeout checking on TX, so we know that's not a factor affecting any packet drop.

EDIT:
I also confirmed that the GENERIC RX code will dump bytes if the sketch doesn't read them as the same rate as the host is sending. It acknowledge every packet back to the host no matter what.

Easy test:
open the port with Serial.begin();
Then loop never reading from the port.
Then send from the host to the host, the host will never block because the MCU will acknowledge all packets even if not reading them.

You can also test by opening the port with Serial.begin().
Then delay for a long enough time.
Then print what's read.
Then from the host send anything longer that the GENERIC buffer size. The host will act like everything was sent, but the MCU will only print what fit in the buffer size. Everything else after that was dumped without the host knowledge.

If you are trying to send something large, like an image, as fast as possible, it's likely that you will lose data unless you sketch can process it faster than the host can send.

From my point of view this is not desirable, since in the host the serial port baud rate has no effect on how fast the application will try to send data, either you implement some handshaking within your applications (in the MCU and the computer) or you never know when the buffer may be full and dumping data.

EDIT2:
I modified the code to behave like libmaple F1, in which if the buffer is full it will not acknowledge packets back to the host. It's in this branch. Is in sync with Daniel other than those changes.
https://github.com/victorpv/STM32GENERI ... imizations

I still need to do further test to confirm there is no corruption with the data anywhere, but I have confirmed that if the sketch is bussy doing something and doesn't pull data from the buffer, it will pause the host when it's full, and resume when it has capacity.

User avatar
Pito
Posts: 1531
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by Pito » Sat Aug 19, 2017 6:32 pm

TX: What if the Serial.write (and friends) will check whether the Host is NAKing, and it timeouts while returning 0 ??
Pukao Hats Cleaning Services Ltd.

victor_pv
Posts: 1654
Joined: Mon Apr 27, 2015 12:12 pm

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by victor_pv » Sat Aug 19, 2017 7:20 pm

Pito wrote:
Sat Aug 19, 2017 6:32 pm
TX: What if the Serial.write (and friends) will check whether the Host is NAKing, and it timeouts while returning 0 ??
You mean without buffering even if there is capacity in the buffer?

So the TX code currently is supposed to wait for NAK before pulling more data from the TX buffer and sending it. And the write() function, as it writes to the buffer and not the USB peripheral, will return 0 if the buffer is full, but not until that point.
Whether this ir working right, or the timeout period is long enough, that's up for debate.
I would favor a timeout that's proportional to what the app is trying to send. Is not the same waiting for buffer space for 1 byte than for 30 or 300.

User avatar
Pito
Posts: 1531
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by Pito » Sun Aug 20, 2017 10:47 am

From user's point of view I do not see any buffers (the internal one in the stm core or in the Host) and I "do not care about what buffers are inside the black box(es)".

I have the Host, which can be unable to receive fast (for any reason).
I do Serial.something (1byte or 1kB or 1MB) from my sketch.

When the Serial.something returns 0, or returns a number which is less than the amount the user has intended to send, it means for the user the data were not "received" by the Host (the other side has not accepted them all, or accepted just a part of it).
Me - the user - I have to care at my sketch level what should happen in such a situation.
In that way you cannot loose any TXed data.

Imagine you are going to upload a picture out of your 100kB array (the array in your sketch, the user_array) via usb Serial to PC Host. It should work such when the PC Host will be able to receive say 1kB per minute via USB you will upload that user_array in 100 minutes successfully to the Host without any data lost..
Pukao Hats Cleaning Services Ltd.

victor_pv
Posts: 1654
Joined: Mon Apr 27, 2015 12:12 pm

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by victor_pv » Sun Aug 20, 2017 4:21 pm

But then you have other people thinking differently:
viewtopic.php?f=51&t=2354&start=30#p33133

The way I see it, we have two options:
1.- Guaranteed delivery with blocking.
2.- Non guaranteed delivery, after a certain timeout fail to send and return the number of bytes that were succesfully sent (or buffered)

But looks like we can't settle on one of the other.
My opcion is 2, and have the application check the return value. That would resemble what happens with a physical USART. The code will not block and just output the data at the given rate.

Also as more advanced feature, we could use the baud parameter that we currently ignore to manage the timeout.
So if we do a Serial.begin(115200), then the timeout should be around 70uS per byte.
That way if you have a slow application to which you need to send no faster than at a 57kbps, you can set the baud rate to that, and the usb write function will block only for enough time for that rate, and return with the number of bytes actually sent if it takes longer than that.

User avatar
martinayotte
Posts: 1222
Joined: Mon Apr 27, 2015 1:45 pm

Re: [STM32GENERIC/HAL] SerialUSB TX/RX speed problem

Post by martinayotte » Sun Aug 20, 2017 4:54 pm


Post Reply