New SdFat library (with and without DMA)

Please do not post requests
stevestrong
Posts: 1603
Joined: Mon Oct 19, 2015 12:06 am
Location: Munich, Germany

Re: New SdFat library (with and without DMA)

Post by stevestrong » Sun Oct 30, 2016 9:40 am

I suggest to use as basis the "RawWrite" example, this is suitable for fast SD card writing, with or without FAT support.
The latest SDFat-beta from greiman should also work, but not tried yet.
I am currently working with a modified version of it, DMA-based double-buffered analog acquisition and storage (while DMA writes the data from ADCs to one buffer, the other buffer being written onto SD card, I think I already posted somewhere an example).
I achieve ~350kBps (4 analog channels, dual simultaneous conversion using ADC1 and ADC2 in parallel, 12bits/channel, 44.1kHz sampling frequency, 1 second) without losing a byte, but without FAT support.
Now struggling to achieve fast read out speed from the card without DMA...
Last edited by stevestrong on Sun Oct 30, 2016 11:04 pm, edited 1 time in total.

SanjuPrakash
Posts: 6
Joined: Thu Oct 27, 2016 7:07 am

Re: New SdFat library (with and without DMA)

Post by SanjuPrakash » Sun Oct 30, 2016 11:08 am

stevestrong wrote:I suggest to use as basis the "RawWrite" example, this is suitable for fast SD card writing, with or without FAT support.
The latest SDFat-beta from greiman should also work, but not tried yet.
I am currently working with a modified version of it, DMA-based double-buffered analog acquisition and storage (while DMA writes the data fro ADCs to one buffer, the other buffer being written onto SD card, I think I already posted somewhere an example).
I achieve ~350kBps (4 analog channels, dual simultaneous conversion using ADC1 and ADC2 in parallel, 12bits/channel, 44.1kHz sampling frequency, 1 second) without losing a byte, but without FAT support.
Now struggling to achieve fast read out speed from the card without DMA...
Could you post the code that you used for the write ? It would be really helpful as I need to log samples at almost the same rate that you mentioned above .

stevestrong
Posts: 1603
Joined: Mon Oct 19, 2015 12:06 am
Location: Munich, Germany

Re: New SdFat library (with and without DMA)

Post by stevestrong » Mon Oct 31, 2016 9:16 am

Here is an older but still working version, good for getting accommodated with the basics.
There is an "issue" in the (older) SDfat lib, because if you want to go without FAT or any other file system, the sd.begin() function returns an error.
You just have to disregard this error, and it will work fine only using cacheClear, writeStart, writeData, writeStop functions (also see the SDFat lib reference documents). This way you write the data directly to card block-wise in an incremental way.
For block-wise incremental raw reading (w/o FAT) I use readStart, readData, readStop functions.
I will soon update the file when I get the upload (from SD card to PC over USB serial) running as fast as possible.

sherinkapotein
Posts: 20
Joined: Thu Apr 06, 2017 6:53 am

Re: New SdFat library (with and without DMA)

Post by sherinkapotein » Fri Apr 14, 2017 10:01 pm

Having come from the Arduino lineup, had always read about how the DMA could improve some mundane but necessary tasks as logging data to SD card. Finally being handed over with a bluePill along with 3V3 SD card module (gawd why are they so tough to get) I'd thought why don't give it a try.
Now that the course of the evening was decided I thought might as well document my findings here.

Following are the results from example file bench of the SdFat library by Bill Grieman.

SD card = SanDisk Class 10 32GB (02/2015)
Size of file to be written = 10MB
Tested with varying buffer sizes from 512B to 16KB (yes I did try that too ) with and without DMA.

The card was formatted after every run using the SD formatter utility developed by SD association and SPI speed set to Full speed.
The figures are averaged out over two runs for each test

As can clearly be seen that DMA does accelerate the process for a sufficiently sized buffer reaching speeds upto 3.2MBPS. Even the implementation without DMA for an 8KB buffer gave 3x performance beyond which the law of diminishing returns kicked in.
sped_wR.PNG
sped_wR.PNG (21.18 KiB) Viewed 630 times
Again supporting the case, we see that is only for larger bufsize that the difference between average write latencies, with and without DMA show some promising trend. For a smaller values of DMA transfer, I presume the overheads associate with invoking and managing a DMA transfer eats into the significant portion of the advantage that one would expect.
avg_wrL.PNG
avg_wrL.PNG (17.8 KiB) Viewed 630 times
The statistics generated by the test code i.e max,average,minimum write latencies weren't too dissimilar initially( initial= testing with smaller bufsize) so I though it would be interesting to get the distribution of the Write latencies grouped under buckets to see if there was anything to take away from that. As for my purpose, any latency below 6ms was good enough I let it go scotfree. Note that graph speaks in percentage of overall transactions. (transactions = FILE_SIZE/BUF_SIZE)
nodma_disT.PNG
nodma_disT.PNG (20.15 KiB) Viewed 630 times

Looks like members can attach 3 images a post; fair enough
<......cont.....>
Last edited by sherinkapotein on Fri Apr 14, 2017 10:21 pm, edited 1 time in total.

sherinkapotein
Posts: 20
Joined: Thu Apr 06, 2017 6:53 am

Re: New SdFat library (with and without DMA)

Post by sherinkapotein » Fri Apr 14, 2017 10:17 pm

dma_disT.PNG
dma_disT.PNG (22.2 KiB) Viewed 629 times
Note the logarithmic scale on Y axis.

So what can be seen is that there is isin't much effect on the distribution of latencies whether using DMA or not. For the graph below the 512B and 2048B buffers show similar distribution of transaction grouped into latency windows for both modes. It is only when, for non DMA mode that as the average write latency increases that the distribution is affected ( my guess is due to having to actually write large chunks of data in case of larger buffers in a blocking mode. <seeks validation>).
fin_disT.png
fin_disT.png (23.62 KiB) Viewed 629 times
What can be concluded is that unless application demands a lot a ram assigning of 8Kb of buffer from the pool for DMA activity should give be a reasonable price to pay for the F103c8t6 (20k Ram) versions of STM32. Anything below 2k for DMA purposes should probably be rather avoided.

If the results will be drastically different for a Class 4 card or other permutation, is something that I leave for another day.
Cheers!!

User avatar
Pito
Posts: 1522
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: New SdFat library (with and without DMA)

Post by Pito » Fri Apr 14, 2017 10:58 pm

I did 3.5MB/sec @36MHz SPI with only 512bytes large buffer. Reported somewhere in this thread..
viewtopic.php?f=13&t=20&start=120#p17911
Pukao Hats Cleaning Services Ltd.

sherinkapotein
Posts: 20
Joined: Thu Apr 06, 2017 6:53 am

Re: New SdFat library (with and without DMA)

Post by sherinkapotein » Fri Apr 14, 2017 11:06 pm

pito wrote:I did 3.5MB/sec @36MHz SPI with only 512bytes large buffer. Reported somewhere in this thread..
Oh I should have known better!!
Off I go to dive into SDFATEX !! :)

User avatar
Pito
Posts: 1522
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: New SdFat library (with and without DMA)

Post by Pito » Sat Apr 15, 2017 8:11 am

You need the latest SdFat-beta and also do mind to enable the EX mode in SdFatConfig.h..
Pukao Hats Cleaning Services Ltd.

stevestrong
Posts: 1603
Joined: Mon Oct 19, 2015 12:06 am
Location: Munich, Germany

Re: New SdFat library (with and without DMA)

Post by stevestrong » Sat Apr 15, 2017 8:13 am

What is EX bringing more than the "normal" beta?

User avatar
Pito
Posts: 1522
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: New SdFat library (with and without DMA)

Post by Pito » Sat Apr 15, 2017 8:17 am

Xtimes faster wr/rd speed.. sherinkapotein will test it :)
Pukao Hats Cleaning Services Ltd.

Post Reply