Non blocking SPI DMA - Added callback to the SPI DMA functions (dmaSend, dmaTransfer...)

Post here first, or if you can't find a relevant section!
victor_pv
Posts: 1681
Joined: Mon Apr 27, 2015 12:12 pm

Re: Planning to add callback to the SPI DMA functions (dmaSend, dmaTransfer...)

Post by victor_pv » Tue Mar 07, 2017 1:52 am

stevestrong wrote:Ok, so you run both SPI ports in DMA mode...
You could try to let only the Rx part in DMA mode, the Tx part in "normal" mode to see if Rx bytes are still lost.
I will test that too.
Today I have tested whennot using callbacks, if I remove the part where the transmission can timeout, it failed with the same situation, so when not using callbacks I was just losing 1 byte here and there too, just didn't block forever thanks to the timeout.

I am starting to see why STM said the ports run at a max 18Mbits in the datasheet.

stevestrong
Posts: 1746
Joined: Mon Oct 19, 2015 12:06 am
Location: Munich, Germany

Re: Planning to add callback to the SPI DMA functions (dmaSend, dmaTransfer...)

Post by stevestrong » Tue Mar 07, 2017 9:23 am

I think the callback should not have any influence on the test result if you wait for TXE and BSY flags within the Tx end ISR.
When checking figures 246 and 247 overlapped, the one bit (2 APB1 clock) gap between RXNE and TXE should give priority to Rx channel if is set to highest priority (11), please double check that you have set the priorities correctly. DMA for Tx can be set to lowest priority (00).

Furthermore, instead of timeout, you could check the DMA_CNDTRx register value to determine whether there are still bytes to be received when Tx finished.

Does it fail when using only SPI_1 both Tx and Rx DMA with 36MHz clock (SPI2 not working)?

If not, then it is clear a race-condition/limitation of the hardware (bus matrix, AHB system bus and the two bridges to APB1 and APB2 peripheral buses as indicated in figure 2) which cannot handle so many data (DMA and CPU<->RAM) transfer requests within the short time period of one byte transfer at 36MHz, as SPI_1 is served on APB2 and SPI_2 on APB1.

Alternatively you could check what happens when setting the flash wait states down to 1 (CPU at 72MHz) or up to 3.

In addition, you could monitor the MODF and OVR error flags of SPI (enable these IRQs?) and/or TEIF of DMA.

EDIT
Can you please specify more details about how exactly do you test? Do you read blocks of 512 bytes on SPI 1 from SD card and read blocks (how many bytes?) from ILI display repeatedly? Do you use the older SdFat lib or the newer one, SdFat beta? I could maybe test in parallel if you share the testing code.

victor_pv
Posts: 1681
Joined: Mon Apr 27, 2015 12:12 pm

Re: Planning to add callback to the SPI DMA functions (dmaSend, dmaTransfer...)

Post by victor_pv » Tue Mar 07, 2017 4:58 pm

stevestrong wrote:I think the callback should not have any influence on the test result if you wait for TXE and BSY flags within the Tx end ISR.
When checking figures 246 and 247 overlapped, the one bit (2 APB1 clock) gap between RXNE and TXE should give priority to Rx channel if is set to highest priority (11), please double check that you have set the priorities correctly. DMA for Tx can be set to lowest priority (00).

Furthermore, instead of timeout, you could check the DMA_CNDTRx register value to determine whether there are still bytes to be received when Tx finished.

Does it fail when using only SPI_1 both Tx and Rx DMA with 36MHz clock (SPI2 not working)?

If not, then it is clear a race-condition/limitation of the hardware (bus matrix, AHB system bus and the two bridges to APB1 and APB2 peripheral buses as indicated in figure 2) which cannot handle so many data (DMA and CPU<->RAM) transfer requests within the short time period of one byte transfer at 36MHz, as SPI_1 is served on APB2 and SPI_2 on APB1.

Alternatively you could check what happens when setting the flash wait states down to 1 (CPU at 72MHz) or up to 3.

In addition, you could monitor the MODF and OVR error flags of SPI (enable these IRQs?) and/or TEIF of DMA.

EDIT
Can you please specify more details about how exactly do you test? Do you read blocks of 512 bytes on SPI 1 from SD card and read blocks (how many bytes?) from ILI display repeatedly? Do you use the older SdFat lib or the newer one, SdFat beta? I could maybe test in parallel if you share the testing code.
I think the callback should not have any influence on the test result if you wait for TXE and BSY flags within the Tx end ISR.
When checking figures 246 and 247 overlapped, the one bit (2 APB1 clock) gap between RXNE and TXE should give priority to Rx channel if is set to highest priority (11), please double check that you have set the priorities correctly. DMA for Tx can be set to lowest priority (00).
The callback has no influence. I verified that by not using callback, and not using timeout to cancel the transfer after a timeout period. Instead it just waits until RX DMA is completed. Doesn't always happen, something there is still 1 RX byte left while TX has finished. Nothing is disabled, there is just not more bytes to send by TX, while RX still shows 1 more transfer left. Since both transfers have the same length, the only explanation I can think off is that 1 byte is getting lost when the DMA should read it.
Furthermore, instead of timeout, you could check the DMA_CNDTRx register value to determine whether there are still bytes to be received when Tx finished.
I did, that's how I found RX had still 1 byte more to receive, CNDTRx for that channels shows 1, the one for TX shows 0, and stays like that forever. If there is no TX, there is no clock to get RX either, so at that point it just blocks waiting forever for that transfer to complete.
We had a timeout in the dmaTransfer function (I dont know if you have it in your version). If I leave the timeout part, it will eventually timeout and cancel the transfer. If I remove the timeout part, it just blocks there waiting for RX to complete, and CNDTRx shows 1 (some times 2, but more often 1).
Does it fail when using only SPI_1 both Tx and Rx DMA with 36MHz clock (SPI2 not working)?
Only fails at 36Mhz on SPI1. At 18Mhz works fine every time, even while still using SPI2 at the same time, both with callbacks (so they can overlap and work at the same time if the transfers are big enough). There is another DMA at 88Khz to play a wav file with PWM in circular mode. I left it running at 18Mhz yesterday night, didn't check this morning but will check later, if it is still running, it did not fail once in 24 hours.

It only fails for RX. Since RX needs TX to generate clock, they are both used. I have not tested yet to use RX with DMA and TX without, I'll test, but since at that point may be slightly slower than 36Mhz, it may not fail. I will need to check for RXNE before sending each byte otherwise I will overwrite them myself. So between checking flags and writing to DR, may not be completely continuous.
Alternatively you could check what happens when setting the flash wait states down to 1 (CPU at 72MHz) or up to 3.

In addition, you could monitor the MODF and OVR error flags of SPI (enable these IRQs?) and/or TEIF of DMA.
Good idea, I will check the OVR flag, didn't think on that. That will help to confirm if a second byte is arriving before the previous is read, which is what I am suspecting so far. Either that, or RX byte in DR is overwritten by TX at which point the SPI device clears the DMA request for the RX channel, so the DMA channel does't read that byte, CNDTR is not decremented, but OVR wouldn't trip. I believe it trips only if the DR is overwritten by an incomnig byte, but not by a write to DR from the MCU. I think TEIF will only trip in case of a serious error, but I can check it to, since I can block waiting for RX, the MCU is pretty much frozen and flags should not change. So I can stop it and look at it.

Another test I have thought, is to build a long set of consecutive numbers in RAM in a buffer, from say 0 to 255 and over again, connect TX and RX pins, fire the DMA and compare the input with the output. That will help to see if a bunch of bytes get lost at once, if it is the first byte, the last, etc...

stevestrong
Posts: 1746
Joined: Mon Oct 19, 2015 12:06 am
Location: Munich, Germany

Re: Planning to add callback to the SPI DMA functions (dmaSend, dmaTransfer...)

Post by stevestrong » Tue Mar 07, 2017 8:44 pm

I don't think that SPI_DR register for Tx and Rx would mix up, the hardware should handle the correct way of data flow of Tx and Rx data to and from SPI_DR. I think it is rather a matter of limitation on internal data buses.

Still, you did not try to run Tx and Rx DMA on SPI 1 with 36MHz and let SPI 2 disabled/inactive. This would give us more information.

victor_pv
Posts: 1681
Joined: Mon Apr 27, 2015 12:12 pm

Re: Planning to add callback to the SPI DMA functions (dmaSend, dmaTransfer...)

Post by victor_pv » Tue Mar 07, 2017 10:13 pm

stevestrong wrote:I don't think that SPI_DR register for Tx and Rx would mix up, the hardware should handle the correct way of data flow of Tx and Rx data to and from SPI_DR. I think it is rather a matter of limitation on internal data buses.

Still, you did not try to run Tx and Rx DMA on SPI 1 with 36MHz and let SPI 2 disabled/inactive. This would give us more information.
There is a single DR register, used for both TX and RX.
If I understand it right, when you write to it, if you read back you will just read what you wrote. I'll try to confirm that from the reference manual, perhaps writing to it does't overwrite the data that was waiting to be read.
When written goes to the TX buffer and when read comes from the RX buffer, so they shouldn't mix as you said, so that can be discarded.

About testing SPI1 at 36Mhz and not using SPI2 at the same time, is easy to test, I'll test it later today.

To answer some of the questions above.
I use sdfat from greyman, I believe the latest stable, but may be using an older version.
Reads from sdcard are 512byte each, but may issue up to 4 consecutively to fill a 1024 bytes buffer. (depending on the file, half the data may be discarded).
The display is an ili9163, uses DMA for sending data to the display, in a different thread than the sdcard, so they may happen at the same time.
Originally both were set to max speed (36 spi1, 18 spi2).
I have tried setting spi1 to 18Mhz, and so far so good, but when I get home I will see if it lasted a day like that or not.
I need to reconfigure github and will upload the whole project there.

stevestrong
Posts: 1746
Joined: Mon Oct 19, 2015 12:06 am
Location: Munich, Germany

Re: Planning to add callback to the SPI DMA functions (dmaSend, dmaTransfer...)

Post by stevestrong » Wed Mar 08, 2017 10:10 am

Victor, how did you adapt the SdFat lib in order to cope with the callback functionality?
Because as far as I know, the SdFat uses currently SPI transfers in a blocking way.

victor_pv
Posts: 1681
Joined: Mon Apr 27, 2015 12:12 pm

Re: Planning to add callback to the SPI DMA functions (dmaSend, dmaTransfer...)

Post by victor_pv » Wed Mar 08, 2017 3:18 pm

stevestrong wrote:Victor, how did you adapt the SdFat lib in order to cope with the callback functionality?
Because as far as I know, the SdFat uses currently SPI transfers in a blocking way.
I am using it with FreeRTOS, so it blocks just that task, using tasks notifications to block and unblock. That task has higher priority than the rest, so as soon as the ISR is called it unlocks it to continue, in the meanwhile the MCU execute some other task.
It just needs a few modifications in the file that contains the SPI functions for the STM32. I'll include that code when I have time to upload to github.
I also tested disabling that part, so no callbacks, and the SPI functions in the SPI library block waiting for complete, and gives the same result, so it is not related to the callback part.

Yesterday I couldn't do much, but I checked this:
-SPI1 OVR flag is set when the RX DMA doesn't finish all the transfers, so the RX value is not always read in time, and OVR ends up tripping.
-Using SPI2 dma or not did not make a difference.

victor_pv
Posts: 1681
Joined: Mon Apr 27, 2015 12:12 pm

Re: Planning to add callback to the SPI DMA functions (dmaSend, dmaTransfer...)

Post by victor_pv » Sun Mar 12, 2017 6:03 pm

I have done a few tests today with a different board, with an F103RCT6 chip.
I connected MISO to MOSI in SPI1, and repeatedly sent and receive with DMA to 2 buffers. Then compare the content.
All was going well when I was only using SPI1, with or without callback.
Then I started using SPI2 also (spi2 without DMA), and the problems start. Some bits are not received correctly, and changes a 0 for 1 in the last bit of some transferred bytes.

The sketch run transfers in a loop and compares the result at the end. If there is an error, it stops and wait for user input, then can repeat. If there is not error it repeats the loop without stopping. When errors happen they happen every few passes, but not every single one. The errors also happen on different bytes, not always the same.

EDIT: I have repeated the same test using SPI2 with DMA, and also setting SPI1 to Div4 speed. These are the results:

DIV2 Speed on spi1:
SPI1 Alone Without callback: OK
SPI1 alone with callback: OK
SPI1 with callback, in parallel SPI2 without DMA: errors
SPI1 with callback, in parallel SPI2 with DMA: errors

DIV4 speed on spi1:
SPI1 with callback, in parallel SPI2 without DMA: OK
SPI1 with callback, in parallel SPI2 with DMA: OK

Seems so far that errors only happen when SPI1 and SPI2 are working at the same time, and SPI1 is operating at 36Mhz (over specs).
I'll try to post all my code to github. I'll stop with this tests here, since it started having issues when I started writting the callback stuff, and didn't know if the problem was on that. I'm convinced now the callback code doesn't have any problem, but the only problem was due to using both SPI ports completely in parallel, which was only allowed by using callbacks to signal the end of transfer.

Conclusion for me: spi1 is not reliable when operating at 36Mhz, specially when using another spi port at the same time. Probably ok for only sending data, or when used alone, or non critical reception.

User avatar
Rick Kimball
Posts: 1038
Joined: Tue Apr 28, 2015 1:26 am
Location: Eastern NC, US
Contact:

Re: Planning to add callback to the SPI DMA functions (dmaSend, dmaTransfer...)

Post by Rick Kimball » Sun Mar 12, 2017 7:46 pm

maybe you could try adjusting the relative priority of the interrupts?
-rick

victor_pv
Posts: 1681
Joined: Mon Apr 27, 2015 12:12 pm

Re: Planning to add callback to the SPI DMA functions (dmaSend, dmaTransfer...)

Post by victor_pv » Sun Mar 12, 2017 10:42 pm

Rick Kimball wrote:maybe you could try adjusting the relative priority of the interrupts?
I could, but since the problem is on a DMA transfer, using DMA for both TX and RX, interrupt priority should not result it corrupted data, should it?
I have tried adjusting the DMA channels priority, but that did not make a different, at least not significant.

Post Reply