Serial buffer

Post here first, or if you can't find a relevant section!
krishna
Posts: 7
Joined: Tue Apr 26, 2016 6:15 am
Location: Hyderabad,India

Re: Serial buffer

Post by krishna » Wed May 04, 2016 10:52 am

WereCatf wrote :
Now, the bigger issue here is that you haven't explained what you're reading from the serial-port, or what you're planning on doing with it -- there are most likely better ways of going about this in the first place
I am just trying to read GPS data.

User avatar
RogerClark
Posts: 7683
Joined: Mon Apr 27, 2015 10:36 am
Location: Melbourne, Australia
Contact:

Re: Serial buffer

Post by RogerClark » Wed May 04, 2016 11:01 am

There are plenty of examples of reading GPS Data

Have you tried the one on the Arduino site

http://playground.arduino.cc/Tutorials/GPS

It looks like it uses a 300 character buffer, into which it reads the GPS chars, until it finds an end marker in which case it processes the data.

User avatar
mrburnette
Posts: 2206
Joined: Mon Apr 27, 2015 12:50 pm
Location: Greater Atlanta
Contact:

Re: Serial buffer

Post by mrburnette » Wed May 04, 2016 11:45 am

krishna wrote:
WereCatf wrote :
Now, the bigger issue here is that you haven't explained what you're reading from the serial-port, or what you're planning on doing with it -- there are most likely better ways of going about this in the first place
I am just trying to read GPS data.
I wished you had said so in first post. I have many examples: https://www.hackster.io/rayburne/projects In one example above, I use Adafruit's GPS library, somewhat hacked :shock:
In one example, the PSoC one, I use my own code to parse the GPS in realtime

This is basic stuff and there are 100's of examples on the Internet. When confronted with a programming problem, go and search for what others have done, pick a couple of good examples of what you want to accomplish, and then dig into the code to understand how someone else used logic to develop a solution. Then, build your own skeleton code, experiment to get the basic functionality, then elaborate and pretty things up if you are so inclined.

Added:
Full code here for Maple Mini
The Adafruit_GPS library double-buffers the GPS serial signal and also inherits from the Serial object, essentially adding methods that makes using the GPS very simple. Object instantiation is like this: Adafruit_GPS GPS(&mySerial);
Then, in the loop() section of the code, the GPS buffer is checked to see if data is waiting, like this:

Code: Select all

SoftwareSerial mySerial(0, 1);
Adafruit_GPS GPS(&mySerial);
//....

void loop()  {
  char c = GPS.read();                                            // read data from the GPS and check for full sentence
  if (GPS.newNMEAreceived()) {
    if (!GPS.parse(GPS.lastNMEA()));                              // sets the newNMEAreceived() flag
  }
Ray

removed code, the entire non-library-based inline parsing code is here for the PSoC 4200 which is easily converted to Arduino'ish syntax.

bluesystems
Posts: 5
Joined: Sun Jan 29, 2017 3:48 pm

Re: Serial buffer

Post by bluesystems » Sat Mar 11, 2017 5:53 pm

I note that on the STM32F103 the serial buffer size does not seem to be pulled from HardwareSerial.h.
It HardwareSerial.h is edited from 64 bytes to 128 there is no change in the serial port's buffer size.

I changed
C:\Users\johnston\Documents\Arduino\hardware\Arduino_STM32-master\STM32F1\cores\maple\HardwareSerial.h

Code: Select all

//#define SERIAL_TX_BUFFER_SIZE 64
//#define SERIAL_RX_BUFFER_SIZE 64
#define SERIAL_TX_BUFFER_SIZE 128
#define SERIAL_RX_BUFFER_SIZE 128
and ran this code with TX and RX pins looped back (joined)

Code: Select all

void loop() 
{
delay(100);
  Serial1.print("A");
  Serial.print(a); Serial.println(" Number of bytes sent"); // char number to dbg
  a=a+1;
    Serial.print("Number of bytes to read : ");  Serial.println(Serial1.available(),DEC);
}
No matter what the buffer size is set to, Serial.println(Serial1.available(),DEC) climbs to 63 and stops, showing the buffer is 63 bytes, the default value.

I am missing some thing here or is the buffer pulled from some other file?
Bob J.

bluesystems
Posts: 5
Joined: Sun Jan 29, 2017 3:48 pm

Re: Serial buffer

Post by bluesystems » Sat Mar 11, 2017 6:35 pm

After doing some digging it become clear the uart buffer sizes in pulled from

C:\Users\johnston\Documents\Arduino\hardware\Arduino_STM32-master\STM32F1\system\libmaple\include\libmaple\usart.h

Changing the file as follows...

Code: Select all

/*
 * Devices
 */

#ifndef USART_RX_BUF_SIZE
//#define USART_RX_BUF_SIZE               64
#define USART_RX_BUF_SIZE               512
#endif

#ifndef USART_TX_BUF_SIZE
//#define USART_TX_BUF_SIZE               64
#define USART_TX_BUF_SIZE               128
#endif
Changes the buffer sizes in usart.h results in the serial port buffer growing to the newly defined sizes.
This is good news for me as in several of my IoT application where I need to be ready at all times to receive a buffer load of data of about 300 bytes from a WEB browser while off busy dealing with some very slow sensors.
No way to control when someone clicks on that WEB link dumping data on me and the sensors are real time sensitive so always pollling for serial data everywhere in the code would be very messy at best and unworkable at worst.
Yes this uses a lot more memory but as the STM32 starts with 20K, that is fine by me.
Bob

asmallri
Posts: 20
Joined: Fri Oct 06, 2017 12:37 am

Re: Serial buffer

Post by asmallri » Sun Jan 14, 2018 3:09 pm

Thanks Bob, This was a great help. I could not find how to fix this ring buffer size problem.

User avatar
mrburnette
Posts: 2206
Joined: Mon Apr 27, 2015 12:50 pm
Location: Greater Atlanta
Contact:

Re: Serial buffer

Post by mrburnette » Sun Jan 14, 2018 9:33 pm

asmallri wrote:
Sun Jan 14, 2018 3:09 pm
Thanks Bob, This was a great help. I could not find how to fix this ring buffer size problem.
Ol' Bob has not been active in the forum since
by bluesystems
Sun Mar 12, 2017 4:14 pm
While it is appropriate to wake-up an old thread to add pertinent information, it is considered bad forum manners to wake up a long sleeping post just to say, "Thank you."

However, his suggestion on enlarging the serial buffer will work, but is not a recommended "fix" since editing the core files is always a last resort.

Rather the coder should implement their own separate buffer(s) if the default size is not satisfactory. If one hacks around in the core, then all changes are overwritten the next time you update... and at the rates changes are being submitted for github pull requests, that could be often.

If you need an example of double-buffering, Lady Ada (Adafruit) has an example here in her GPS parser:

https://raw.githubusercontent.com/adafr ... it_GPS.cpp

Ray

stevestrong
Posts: 2061
Joined: Mon Oct 19, 2015 12:06 am
Location: Munich, Germany
Contact:

Re: Serial buffer

Post by stevestrong » Sun Jan 14, 2018 10:41 pm

As Ray said, you should not change the buffer size in the core.

If your sensor is reeeeaaaaallllly slow, then I would recommend to attach a short function to the systick interrupt which (each 1 ms) will read a character from the serial, if received, and stores it in an additional larger buffer. (viewtopic.php?f=18&t=2117)
This way you don't have to mess with the core inner organs.

asmallri
Posts: 20
Joined: Fri Oct 06, 2017 12:37 am

Re: Serial buffer

Post by asmallri » Mon Jan 15, 2018 9:12 am

Thank you for the advice, I understand exactly what you are saying however in my case large interrupt driven receive ring buffers are the appropriate solution.

It is unfortunate that it requires modding the core but there are some things, such as the ability to specify the size of ring buffers, do not belong in the core. It should be possible to specify the size of a ring buffer per device without having to perform your own management of two tier ring buffers by performing unnecessary buffer to buffer copies.

stevestrong
Posts: 2061
Joined: Mon Oct 19, 2015 12:06 am
Location: Munich, Germany
Contact:

Re: Serial buffer

Post by stevestrong » Mon Jan 15, 2018 9:23 am

asmallri wrote:
Mon Jan 15, 2018 9:12 am
It should be possible to specify the size of a ring buffer per device without having to perform your own management of two tier ring buffers by performing unnecessary buffer to buffer copies.
Of course it is possible by using dynamic allocation, but not preferred, as malloc() could by case behave other than expected, increase the generated bin size, also dependent on the compiler version and used compiler switches. For relatively small RAM (embedded systems) the dynamic allocation brings no advantage, just increases the risk the application to fail.
This has been discussed already and has been decided that static size definitions will be used.
It was agreed that there are other methods (like those suggested) which help when larger buffer is needed.

Whether the ring buffer is interrupt driven or not is of lower importance if you have anyway slow sensors which hinder you to get the received data from the serial interface, which forces you to increase the size of the buffer.

Post Reply