SDcard Logger with FIFO and FIR Filter (FreeRtos)

Post here first, or if you can't find a relevant section!
User avatar
Pito
Posts: 1269
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

SDcard Logger with FIFO and FIR Filter (FreeRtos)

Post by Pito » Fri Oct 07, 2016 5:45 pm

SDcard Logger with FIFO and FIR Filter (FreeRtos) for Maple Mini or Blue Pill
==================================================================
As it has been discussed in this forum, writing a continuous stream of data to an SDcard is difficult.
Not because of the SDcard's writing speeds, but because of Write Latencies (up to 250ms).
During an WL an SDcard does not respond to the incoming data, it means when the data are not buffered, the data are lost.
You have to overcome the WL outages (random occurence, frequent, random duration) somehow, here is an example with a "FIFO buffering".
More on this WL issue you may find here:

http://www.stm32duino.com/viewtopic.php ... 288#p18207

Below you may find a complete Demo I wrote for MapleMini and an SDcard connected to SPI1:
1. we sample 8ADCs, 200x per second,
2. we apply 30Hz low pass filter on each ADC channel measurement,
3. we write the data into the FIFO buffer,
4. in parallel we read the data from the FIFO and write the data onto an SDcard as fast as we can,
5. the format we write the data on the Sdcard is CSV, so you may analyze what is going on in Excel,
6. we also log information on FIFO statuses (the values of counting semaphores).

Note:
1. In this demo we read 8ADC channels as an example, but some ADC channels read SPI1 signals,
as you may see on the results in the CSV file,
2. You may play with FIFO size in Records (here default 150), the actual content of a Sample Record, and with
Sample period/rate (default 5ms =200Hz) such it all fits into MapleMini memory (we assume you got 20kB ram)
and you do not get FIFO overruns when an WL hits,
3. I've tried to comment the code as the time permitted,
4. Provided As-Is, no warranties of any kind,
5. Use at your own risk.

Enjoy logging,
Pito
Attachments
STM_SDLOGGER_FreeRtos_FIFO__FIRFILTER.zip
(4.84 KiB) Downloaded 159 times
Pukao Hats Cleaning Services Ltd.

Sandrine
Posts: 15
Joined: Tue Jun 28, 2016 8:06 pm

Re: SDcard Logger with FIFO and FIR Filter (FreeRtos)

Post by Sandrine » Sun Oct 23, 2016 5:12 pm

Awesome work! I mean great that the .ino is laid out in a way that is understandable.
As you seem to know the system, I have a 16 mega-word buffer that could be treated as a FIFO. Would this work or is the SD Card going to hiccup before then? You mention 20K, have you tried dumping that much? Just trying to understand the buffer sizes ;)

User avatar
Pito
Posts: 1269
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: SDcard Logger with FIFO and FIR Filter (FreeRtos)

Post by Pito » Fri Oct 28, 2016 12:43 pm

The FIFO buffer size:
================================
FIFO_size = data_rate * max_expected_sdcard_write_latency

Example:
==============
You sample 100bytes 500x per second.
data_rate = 100 x 500 = 50kB/sec (this is the speed you want write data to the sdcard).
max_expected_sdcard_write_latency = 250msec

FIFO_size = 50kB/sec * 0.25sec = 12.5kB
Would this work or is the SD Card going to hiccup before then? You mention 20K, have you tried dumping that much? Just trying to understand the buffer sizes
Mind the writing the data to the FIFO AND to the Sdcard is done "in parallel" - thus it does not matter how the FIFO is large (provided it has been designed large enough for the given purpose) :)
You do not "dump" the FIFO in a single shot - it is a continuous process done in parallel - you pushing the data into FIFO AND data from FIFO into the Sdcard at the "SAME" time.. Look at FifoFree and FifoData in the resulting .csv file - when an WL occurs the FifoFree starts to decrease and FifoData increases (sample after sample) - after the WL is over, the process reverses itself..
Try the example/sketch source above - it uses almost all ram available. Also mind in the sketch it uses FIFO size of 150 "records of data" where the record consist of many items, and it writes the items as a READABLE_TEXT into the .CSV file - thus the datarate is the amount of TEXT CHARS (inclusive commas, spaces, \r\n) plus other additional info written to the Sdcard per second..
Also read this topic: http://www.stm32duino.com/viewtopic.php ... 288#p18207
Pukao Hats Cleaning Services Ltd.

Sandrine
Posts: 15
Joined: Tue Jun 28, 2016 8:06 pm

Re: SDcard Logger with FIFO and FIR Filter (FreeRtos)

Post by Sandrine » Tue Nov 01, 2016 2:21 am

Thankyou for your reply,

What I really meant was how much can one stuff into the SD Card buffers before that WL must happen?
I can re-write for the large external FIFO but need to know if it's useful or not. I need to dump about 2M/second

-edit- OK I just saw in the other thread you had mentioned >2Megs / WL so I guess it's worth to pursue
So with a very large RAM / FIFO buffer, continuous streams at 500K/S could easily be realized by streaming to the
SD card at, say 750K/S so the take-up max time of 250mS won't ever be an issue.

Thanks!!

..But there must be a limit on the amount of data the SD Card can hold before transferring it in the flash cells...

User avatar
Pito
Posts: 1269
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: SDcard Logger with FIFO and FIR Filter (FreeRtos)

Post by Pito » Tue Nov 01, 2016 2:51 pm

..1) ..how much can one stuff into the SD Card buffers before that WL must happen?
-edit- OK I just saw in the other thread you had mentioned >2Megs / WL so I guess it's worth to pursue
2) So with a very large RAM / FIFO buffer, continuous streams at 500K/S could easily be realized by streaming to the
SD card at, say 750K/S so the take-up max time of 250mS won't ever be an issue.
3)..But there must be a limit on the amount of data the SD Card can hold before transferring it in the flash cells...
1) No idea, it depends on the actual SD card, the WL is a "random" effect..
2) The max data streaming rate depends on the MCU performance, SDcard type, SDcard interface parameters - ie. SPI clock freq., type of interface - SPI or SDIO, etc.
3) It depends on the actual SDcard - the card's HW differ, firmware differ as well, so hard to answer..
The best way is to experiment and measure.. :)
Pukao Hats Cleaning Services Ltd.

Sandrine
Posts: 15
Joined: Tue Jun 28, 2016 8:06 pm

Re: SDcard Logger with FIFO and FIR Filter (FreeRtos)

Post by Sandrine » Tue Nov 01, 2016 4:23 pm

Thankyou so much Pito, you've been a great help!
WL is random makes sense now, guess it's off to the scientific experiments

Cheers,

Sandy

michael_l
Posts: 336
Joined: Mon Aug 24, 2015 6:11 pm

Re: SDcard Logger with FIFO and FIR Filter (FreeRtos)

Post by michael_l » Wed Nov 09, 2016 10:30 am

Pito, can you suggest some good but not too expensive SD Card adapter from ebay for use with BluePill ? I have some SPI ILI9341 TFT displays with SD CArd holder. Are they OK ?

EDIT: Found one with buffered chip:

http://www.ebay.co.uk/itm/Micro-SD-Stor ... 1033157413

User avatar
Pito
Posts: 1269
Joined: Sat Mar 26, 2016 3:26 pm
Location: Rapa Nui

Re: SDcard Logger with FIFO and FIR Filter (FreeRtos)

Post by Pito » Wed Nov 09, 2016 1:38 pm

When using an SDcard with MapleMini or BluePill avoid the SdDcard's shields/boards with level shifters, voltage translators, buffers/drivers, resistors in the signal paths.
The whole setup has to be 3.3V straight.. The only thing which is important is the SDcards are 3.3V, so a voltage regulator is needed when the SDcard is powered from 5V, and good decoupling (between VCC3V3 and GND) close to the SDcard (ie 2x 10-22uF ceramic multilayer).
Pukao Hats Cleaning Services Ltd.

michael_l
Posts: 336
Joined: Mon Aug 24, 2015 6:11 pm

Re: SDcard Logger with FIFO and FIR Filter (FreeRtos)

Post by michael_l » Mon Feb 13, 2017 2:29 pm

This is the closest I could find based on your recommendation. Quite many seem to have LDO regulator. This one does not but it still has some resistors. Can you recommed this ? Thanks.

http://www.ebay.co.uk/itm/LC-Technology ... OSwARZXoem~

User avatar
BennehBoy
Posts: 420
Joined: Thu Jan 05, 2017 8:21 pm
Location: Yorkshire
Contact:

Re: SDcard Logger with FIFO and FIR Filter (FreeRtos)

Post by BennehBoy » Mon Feb 13, 2017 2:47 pm

One of the future requirements of the project I'm working on will be to dump data out to SD on a continual basis, the sampling frequency will be much lower than this, roughly once every 250ms, so in real terms I could write 4 times as much data once per second if the write/SD overheads were high.

There will be a maximum of 20 readings to be output.

So, question, am I likely yo be impacted by this or is my output volume 'under the threshold'?
-------------------------------------
https://github.com/BennehBoy

Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 1 guest