Serial-only USB boot loader

STM32duino bootloader aka Maple bootloader
Post Reply
jcw
Posts: 171
Joined: Mon Oct 26, 2015 8:16 am

Serial-only USB boot loader

Post by jcw » Sun Nov 01, 2015 3:51 pm

Has anyone explored this serial-only USB boot loader, as alternative for the DFU approach?

original repository: https://github.com/leaflabs/maple-bootloader
further developments, but still relatively old: https://github.com/j1rie/maple-bootloader

The interesting bit is that this might avoid the DFU vs VCP (serial) complexity. In particular, it looks like this approach avoids the need for any driver setup and config files on Windows, Mac OSX, and Linux, and that even pulsing D+ for a reattach may no longer be necessary, since the USB connection remains of the same type. Lastly, the use of the Atmel STK protocol in this driver means that "avrdude" might be all that's needed to upload firware.

If all of this is indeed the case, it could be a very interesting avenue to explore IMO, but for some reason there seems to be no recent development going on in this area, at least not any I could find.

-jcw

Uodate - just found this thread: http://www.stm32duino.com/viewtopic.php?f=32&t=31
Seems related. Reading now. Apologies for the noise. Film at eleven.

jcw
Posts: 171
Joined: Mon Oct 26, 2015 8:16 am

Re: Serial-only USB boot loader

Post by jcw » Tue Nov 03, 2015 11:00 am

The exploration continues. I'd like to try extending or implementing a CDC ACM based serial USB boot loader.
This would jump to the loaded sketch, which continues to use the same driver - avoiding reset / handover issues.

Have been looking at libopencm3 as starting point, but as far as I can tell it only does USB in polling mode?
That would rule it out for sketches in the IDE. The demos from libopencm3-examples do work out of the box.

There's the current libmaple implementation, but I'm not sure what its status is.
Working, clearly, but not using recent STM headers and conventions, it seems?
For future consolidation, using the same base code for all of the IDE runtime code would be nice.

There are a lot of projects for STM32 and USB out there :) - if anyone has a suggestion for a good starting point for a small solid interrupt-driven USB serial implementation for STM32's (F1 for now), please tell...

-jcw

User avatar
mrburnette
Posts: 1803
Joined: Mon Apr 27, 2015 12:50 pm
Location: Greater Atlanta
Contact:

Re: Serial-only USB boot loader

Post by mrburnette » Tue Nov 03, 2015 1:23 pm

jcw wrote:The exploration continues. I'd like to try extending or implementing a CDC ACM based serial USB boot loader.
This would jump to the loaded sketch, which continues to use the same driver - avoiding reset / handover issues.
<...>
Generally, the development of the bootloader is a separate entity from the user sketch design; which is to say that sharing code between the two code bases is not usually done. Previous research in this area indicates it can be done, but making the Arduino linker script understand that their is code in the bootloader section that is reusable is not documented; at least from late last year.

The above is the reason that the Arduino 32U4 devices have the USB serial code linked into the USER script.

Ray

jcw
Posts: 171
Joined: Mon Oct 26, 2015 8:16 am

Re: Serial-only USB boot loader

Post by jcw » Tue Nov 03, 2015 3:10 pm

Ray,

My thought was: low 8 KB flash and low 2 KB RAM are for the serial USB boot loader. Uploads write to the flash memory above that.

Sketch is compiled with flash offset 0x2000 and RAM offset 0x0800, it doesn't contain a USB driver, and doesn't touch the USB hardware. Instead, the boot loader has a vector table in a known place, which the sketch then uses to call into, for plain character-based serial I/O over USB.

With interrupts, the sketch vector entries for USB (3, as I understand) will need to point to the same boot code once re-vectored.

So as far as USB is concerned, there is no interruption of service between when the boot loader ends (with inited and working USB device) and the sketch starts - as there's no µC or device reset in between. The USB driver in the boot loader can re-use the LeafLabs Maple trick for detecting a special "self-reset" request.

It's mostly conjecture and fantasy so far. But it would be nice find out ASAP if such an approach is flawed in any way.

-jcw

User avatar
mrburnette
Posts: 1803
Joined: Mon Apr 27, 2015 12:50 pm
Location: Greater Atlanta
Contact:

Re: Serial-only USB boot loader

Post by mrburnette » Tue Nov 03, 2015 3:52 pm

jcw wrote:Ray,

My thought was: low 8 KB flash and low 2 KB RAM are for the serial USB boot loader. Uploads write to the flash memory above that.

Sketch is compiled with flash offset 0x2000 and RAM offset 0x0800, it doesn't contain a USB driver, and doesn't touch the USB hardware. Instead, the boot loader has a vector table in a known place, which the sketch then uses to call into, for plain character-based serial I/O over USB.
<...>
Unless I am missing something, my point was to the "C" initialization code for the sketch ... it has no clue about the firmware routines; therefore, that area is not initialized; the symbol table for the bootloader is not available to the sketch linker. The user sketch has no clue of the typedefs utilized in the bootloader ... it is a mess, IMO.

This can be done based upon research, but it is not going to be straightforward.

Ray

jcw
Posts: 171
Joined: Mon Oct 26, 2015 8:16 am

Re: Serial-only USB boot loader

Post by jcw » Tue Nov 03, 2015 4:05 pm

That's what the vector table is for - pointers which allow indirect calls to the boot loader code.

Here's an example from NXP for LPC µCs, used in one of my projects: https://github.com/jeelabs/embello/blob ... #L156-L170 - the struct defines pointers to code in ROM, and that struct is located at a fixed known address.

To put it differently: the hookup is done at run time, not at compile/link time.

User avatar
RogerClark
Posts: 6877
Joined: Mon Apr 27, 2015 10:36 am
Location: Melbourne, Australia
Contact:

Re: Serial-only USB boot loader

Post by RogerClark » Tue Nov 03, 2015 8:04 pm

Jean-Claude

I must admit, I'm still not sure why you want a serial bootloader, or one that doesnt re enumerate the USB?

Is it for use with serial OTA updates.

Although the current bootloader is not perfect, most people seem to be able to use it.
(I know you have problems with it on OSX, but I know it works for many people on OSX, so I think you have a specific config problem with your Mac).

BTW. To use OTA
I think its better to use an external processor / radio module, e.g. ESP8266 or nRF51822, then the STM32 could be flashed using SWD by the co-processor, or even by using the internal serial protocol hardware bootloader.

But I know the range on BLE and Wifi is not as good as 433MHz.
However you could use the nRF9E5, as I think someone has this working with an open source compiler. (I know the 9E5 is an obscure device and not used much, but perhaps someone else makes a 433MHz device with MCU)

jcw
Posts: 171
Joined: Mon Oct 26, 2015 8:16 am

Re: Serial-only USB boot loader

Post by jcw » Tue Nov 03, 2015 11:14 pm

It'll all become clearer I hope once the code exists, and I can describe the workflow this leads to.

No, this is not for OTA uploads via wireless, which are indeed on my radar. That will also need a boot loader, but I'll be using a completely different mechanism for that. That code exists, but still needs to be adapted to the STM32. For now, I first need to learn how boot loaders on STM32 work, and try to understand enough (but no more) of USB to get things working.

It looks like the libopencm3 USB driver (same as used in the BMP) can probably be used in interrupt mode after all, so I'm going to stick with that for now.

User avatar
RogerClark
Posts: 6877
Joined: Mon Apr 27, 2015 10:36 am
Location: Melbourne, Australia
Contact:

Re: Serial-only USB boot loader

Post by RogerClark » Wed Nov 04, 2015 12:34 am

It looks like the libopencm3 USB driver (same as used in the BMP) can probably be used in interrupt mode after all, so I'm going to stick with that for now.
Libopencm3 is much more up to date than the code the the current bootloader uses, so I think its a good choice

jcw
Posts: 171
Joined: Mon Oct 26, 2015 8:16 am

Re: Serial-only USB boot loader

Post by jcw » Thu Nov 05, 2015 12:01 am

I'm reporting this here, though it touches on another thread, about having to call Serial.begin() in the sketch - viewtopic.php?f=3&t=703

Yes, serial USB has to be running & attached for it to pick up a reset request (I'll call it "The Maple Trick"). This may also imply that anyone using a different USB mode such as HID or MEM disk will run into this same issue.

This is one of the issues an all-serial-non-re-enumerating boot loader can solve, I hope - with the help of the IWDG watchdog:

* on reset, the boot loader sets up USB serial
* when the sketch runs, it continues to use the boot loader's USB driver
* we set up a watchdog and trigger it main, just after each call to loop() for example
* a stuck sketch will thus reset, maybe even periodically
* each time, the boot loader will start and re-init USB serial, and wait for boot requests for some time
* the serial USB driver continues to implement The Maple Trick

What this would do, is to keep the board entering USB serial mode every once in a while, even if the sketch crashes. With a watchdog timeout of 5 seconds, and with the boot loader waiting at least one second for incoming upload-/reset-req's, that means the board should be able to replace a faulty sketch at least 20% of the time. Finally, if the uploader on the PC side were to keep trying for at least 6 second, it would be guaranteed to succeed.

There are still failure modes where this may be insufficient (keeping the watchdog triggered while the USB driver isn't functioning properly), but hopefully this would be considerably less likely. In that ultimate case, the perpetual boot mode trick would still be needed, but it could be a manual jumper-wire action, as last resort.

The two key points which will make this work, are: 1) that the boot loader gets control first and cannot easily be damaged (as it's not part of uploaded code), and 2) that the watchdog reset kicks in for sketches which are in serious trouble (e.g. crashed, interrupts disabled, infinite loop).

All of this is theory for now. First, there has to be a working serial boot process. I'm still many steps away from that.

Post Reply