Creating a Bootloader for generic boards

Bootloader for boards that don't have the addition hardware found on the Maple mini, which resets the USB
Post Reply
User avatar
RogerClark
Posts: 5974
Joined: Mon Apr 27, 2015 10:36 am
Location: Melbourne, Australia
Contact:

Creating a Bootloader for generic boards

Post by RogerClark » Wed Apr 29, 2015 7:40 am

In the Ideas and Suggestion section, @zoomx suggested that a bootloader for Generic boards would be worth investigating

The difference between the Maple mini and Maple boards and the generic boards, is that the Maple boards have the additional hardware to reset the USB and force the host system (PC etc), to look again at the USB ID of the board.

The Maple boards need to reset the USB, because the upload protocol used is DFU, and requires the board to identify on USB as a DFU device when it boots.

However the Serial USB that the Maple mini has, is built into the sketch, and requires a different USB device ID code VID/PID. So to force the host system (pc etc) to recognise that the board is a serial device when the sketch runs, the board has to reset the USB bus, which is does by pulling one of the USB lines high (possibly to 5V) using a small circuit consisting of two transistors.


If the USB device, didn't need to change from DFU for upload, to Serial for the sketch, there would be no need to reset the USB bus, and hence the board would not need the reset hardware.

It looks like LeafLabs started to develop a version of the bootloader that we may be able to make use, because it uploads via serial, and it uses the same USB VID/PID as the sketch

See https://github.com/jonatanolofsson/mapl ... erial-boot

There is an extensive description, which seems to imply that with modification to the Sketch there would be no need to reset.

So its definitely something worth investigating, so I will probably re-organise the repo so that we have 2 different bootloaders rather just the DFU upload one we currently use.

bobc
Posts: 20
Joined: Mon Apr 27, 2015 11:20 pm

Re: Creating a Bootloader for generic boards

Post by bobc » Thu Apr 30, 2015 10:31 pm

I had a look through the "new" LeafLabs proposal, and I am sorry to say that I don't think it will help much.

Some background: my day job includes writing bootloaders for embedded micros :) In the past, these have been connected by simple serial (RS232 or RS485) with a UART peripheral. For our embedded products (no external access) we have developed a quite robust and easy to use bootloader. It does have one potential weakness, but so far it has not caused a problem, as we carefully manage and test software releases.

Recently though, I ported our "standard" bootloader to a new product using a native USB. I encountered the same problems that LeafLabs (and others) ran into with the interaction between the device, USB and Windows. I got it to work ok, but it was not as good as the simple UART version. For us, downloads are an occasional thing, so it didn't matter if it was a bit clunky.

Our experience with native USB (and other problems, e.g. drivers!) persuaded us to drop that route, and in our next product use an onboard FTDI USB-serial chip to do the USB, feeding it into a UART on the micro. This works a lot better, and I was able to port the bootloader in a couple of weeks.

So anyway, enough about me... What I learned from all that was to avoid bootloader and application sharing the USB. There are some systems that implement USB bootloader in quite a seamless way (without extra helper chips, like mbed), for example Smoothieboard and R2C2 printer controllers (I had a hand in the latter). Those were made easier by some features of the LPC1768, which has a built in "drag and drop" bootloader (nice feature).

Getting a truly seamless bootloader that works well in all case probably requires an external helper chip, and a specific hardware/software design. The Teensy 3.1 works really well in that respect. If we are not working with our own hardware or software design, it will need some compromises.

Perhaps we could define some use cases, and develop a bootloader that covers most of them. I realise most of this post is generalities, I will have a mull over and come back with some specifics.

User avatar
RogerClark
Posts: 5974
Joined: Mon Apr 27, 2015 10:36 am
Location: Melbourne, Australia
Contact:

Re: Creating a Bootloader for generic boards

Post by RogerClark » Thu Apr 30, 2015 10:53 pm

Bob,

OK. Thats a shame

I thought that perhaps because the Serial Bootloader, only enumerated as a Serial device, that perhaps we could avoid the problems with them multiple devices on Windows.

Actually, I think there is some issue with the Serial USB that gets built into the sketch.

I changed the repo last week, so that Serial USB is compiled in, on a separate flag to the bootloader flag, in fact i removed the -DBOOTLOADER_maple as it did too many unrelated things

and added -DSERIAL_USB instead

I changed one of the board definitions (the F103ZET) to have have an option where the bootloader was compiled in, and low and behold a generic STM32F103ZET board, gets USB Serial like the Maple

However.....

I was testing uploading via STLink last night, and I have realised that this doesn't really work properly

If I upload, then physically disconnect and then reconnect the USB connector, the board appears as a Maple serial device (using the windows serial driver)
and I can use Serial.println() etc

However if I upload via STLink, which runs the code again, - although Windows still reports the Serial device, the text doesnt appear in the terminal :-(

I presume that what must be happening is that the virtual serial driver on windows is getting out of sync with whats happening in the serial USB code in the sketch

So I'm going to remove this as an option until we can come up with a fix.

Or perhaps there is no software fix for this?

Do you know if there is anything that the serial code in libmaple could additionally do, to let the driver in windows know its been rebooted ?

Or perhaps STLink needs to reset the whole board, rather than just running code at 0x800000 (though I dount that would make any difference )

Thanks

Roger

mrbwa1
Posts: 91
Joined: Mon Apr 27, 2015 3:36 pm
Location: Buhl, ID
Contact:

Re: Creating a Bootloader for generic boards

Post by mrbwa1 » Fri May 01, 2015 3:06 pm

Roger,

Please forgive my ignorance here, but is the generic implementation releasing the port and is there something in the sketch that is waiting for windows to re-enumerate serial devices.

I don't have STM32 hardware yet, but the ATMega32u4 based boards have issues with serial as well since the 32u4 is handling USB for upload AND serial communications.

http://www.arduino.cc/en/Serial/IfSerial

It took me a good while to get serial to work right on the Micro 32u4 even though it;s simple code in the sketch.

You probably have been through this, so please disregard if you have,. It just sounds like the same thing the Leonardo/Micro does after uploading a sketch.

User avatar
RogerClark
Posts: 5974
Joined: Mon Apr 27, 2015 10:36 am
Location: Melbourne, Australia
Contact:

Re: Creating a Bootloader for generic boards

Post by RogerClark » Fri May 01, 2015 9:35 pm

In the existing bootloader, the USB device for upload (DFU protocol) is completely different from the USB device for Serial when the sketch runs.

Leaflabs have an excellent description of how the Maple bootloader works.
See http://leaflabs.com/docs/bootloader.html

Basically, they initially tried to have the DFU and Serial USB device firmware in the bootloader, and for the bootloader code to continue to execute after the sketch was running ( as the bootloader code would handle serial USB for the sketch)

But basically that would not work on windows.

So the USB serial code was moved to the sketch.
Hence its possible to build a sketch for any generic board that has USB serial.
I have tried this on my stm32f103ZET and F103RCT boards and they do appear as serial devices on my PC and I can use Serial.print etc

However the problem is that every time you upload the sketch to a generic board, you then have to unplug and reconnect the USB otherwise windows doesn't now the board has been reset and the serial driver seems to get out if sync with the board.

And the same applies to the Maple bootloader. After the upload (using the DFU protocol), the bootloader shuts down all its USB code, and jumps to the start of the sketch in Flash memory.
However if the USB bus is not reset, the PC doesnt notice that the DFU device has been removed and that the board is now a Serial device.

To get around this problem, Leaflabs added a small circuit, that pulls one of the USB lines high, which the host PC notices, and the PC checks what device is now connected .


If you look at the Serial branch of the Maple bootloader, I linked to earlier in this thread, This version of the bootloader always stays as a serial device, so technically, there is no need to reset the USB bus to signal a change of device to the PC , when the code in the sketch runs.

But... My experience with trying serial USB on generic boards has highlighted things are unlikely to be plain sailing, and as BobC has said, its very very hard to produce good bootloaders.

I will make a separate folder with the Maple Serial bootloader, from the Serial bootloader branch, so that we have some code for people to work with, if they feel like playing with it.

I will also post instructions on how to compile the bootloader under Windows as to use Make you need to install MinGW.

You can of course use CooCox or Em::Blocks to compile the bootloader, but Im not sure I will have time to setup an Em:Blocks project

mrbwa1
Posts: 91
Joined: Mon Apr 27, 2015 3:36 pm
Location: Buhl, ID
Contact:

Re: Creating a Bootloader for generic boards

Post by mrbwa1 » Fri May 01, 2015 10:48 pm

Interesting implementation on the Maple boards. It would be cool if you could signal the USB line with code to pull high, but I'm thinking that would still require hardware.

Not a big surprise to have to cut power and power it back up to get serial to work. Too bad we don't have a link to the cpChinese manufacturers to request the Maple hardware to pull the line high.

User avatar
RogerClark
Posts: 5974
Joined: Mon Apr 27, 2015 10:36 am
Location: Melbourne, Australia
Contact:

Re: Creating a Bootloader for generic boards

Post by RogerClark » Fri May 01, 2015 11:46 pm

Too bad we don't have a link to the cpChinese manufacturers to request the Maple hardware to pull the line high.
The Maple mini clones have the hardware already, its just the generic boards that don't

But there are loads more generic board types, and the Maple minis are all F103CB's

It think only iTead studio sell a Maple board ($13 on special offer), but it uses the F103RB chip not the F103RE, hence it doesn't have the DAC's. so you may as well buy a generic F103VE for around the same price, as it will have DACs and more flash and I think more RAM

The idea behind producing a Bootloader for generic boards, was that no additional hardware would be required, so everyone could use any of the dozens of generic boards that are coming out of China

Anyway, I'll add the Serial Bootloader to the repo and call it Generic bootloader, and leave it for the community to look at possibilities

mrbwa1
Posts: 91
Joined: Mon Apr 27, 2015 3:36 pm
Location: Buhl, ID
Contact:

Re: Creating a Bootloader for generic boards

Post by mrbwa1 » Sat May 02, 2015 1:10 am

I'm still,getting the hang of these boards. You and a couple others pointed me to the maple mi is vs the generic F103C8 boards.

Maybe I'll have a go at the generics next... Or a discovery STM32F4 board... So many choices!

User avatar
RogerClark
Posts: 5974
Joined: Mon Apr 27, 2015 10:36 am
Location: Melbourne, Australia
Contact:

Re: Creating a Bootloader for generic boards

Post by RogerClark » Sat May 02, 2015 3:04 am

@mrbwa1

Support for the F4 is very limited at the moment. The status of the port is roughly equivalent to the Arduino API from 3 or 4 years ago.

I'd not recommend you get an F4 board, unless you fancy updating the core code ;-)

If you don't have a Maple mini, see MrBrunettes recommendation on who to by from.

If you already have a Mape mini and want a board with a bit flash and ram, I think the most cost effective solution are the STM32F103VET boards available on AliExpress.

For Generics you'd need a USB to Serial adaptor and you may want to consider getting an STLink (clone) for a few $.

User avatar
RogerClark
Posts: 5974
Joined: Mon Apr 27, 2015 10:36 am
Location: Melbourne, Australia
Contact:

Re: Creating a Bootloader for generic boards

Post by RogerClark » Sun May 03, 2015 10:48 am

I just came across this interesting snippet of information

http://stackoverflow.com/questions/5297 ... ly-removed

I'd need to figure out exactly what code to write, but basically this is saying that its possible to force Windows to think that the USB device has been disconnected and reconnected.

This is what the extra hardware does on the Maple mini board

@hull already has this working on Linux using this code

Code: Select all

/* usb-reset -- send a USB port reset to a USB device

    Compile with ...  
    gcc usb-reset.c -o usb-reset
    ... then copy the resulting usb-reset binary to /usr/bin or some other suitable place in your PATH

*/

#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/ioctl.h>

#include <linux/usbdevice_fs.h>


int main(int argc, char **argv)
{
    const char *filename;
    int fd;
    int rc;

    if (argc != 2) {
        fprintf(stderr, "Usage: usbreset device-filename\n");
        return 1;
    }
    filename = argv[1];

    fd = open(filename, O_WRONLY);
    if (fd < 0) {
        perror("Error opening output file");
        return 1;
    }

    printf("Resetting USB device %s\n", filename);
    rc = ioctl(fd, USBDEVFS_RESET, 0);
    if (rc < 0) {
        perror("Error in ioctl");
        return 1;
    }
    printf("Reset successful\n");

    close(fd);
    return 0;
}
I didnt think the same thing was possible on Windows, but if it is, we may be able to write an uploader exe that resets the board, calls DFU-Util to upload the code, and then simulates a USB reconnection.


In which case we can use the bootloader without any changes to the hardware of the bootloader, the change would be to the uploader code

Edit.

There appears to be a Windows utility called decon.exe which can be used to restart a USB device.

I downloaded a copy, but its only the 32 bit version and I'm running 64 bit windows 7, so I need to try the 64 bit one, which it looks like I'll need to extract from the Windows Device Dev Kit.
Fortunatly I think I have a copy of the WDK I downloaded for some other work, so I'll see if I can extract the file and try it, and report back to the forum

PS. Even if that works, I'm not sure how I promote a exe to always run as admin, as I'm pretty sure it is required to run as admin to do this action

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest