Frequent programming defects - bootloader problem?

[D3thAdd3r]

You are getting some amazing (and doing so quickly) development done all around. I will happily sponsor you as many pre-programmed '644s as you need if you run into problems, to keep your going. I doubt you will brick one though, you basically have to screw the fuses into a clock configuration that you don't have hardware to support, but really should have no need to touch fuses.

[Jubatian]

SD card problem!

I started with writing a very simple test program (or rather a pair of programs) to test the bootloader with. It is just an 56KByte pattern with an as minimal Mode 5 Uzebox program as possible to test that pattern's integrity. It can not detect addressing flaws at all (the pattern isn't suitable for it), only data flaws assuming they are (or were attempted to be) written in their proper place. Still for me it did show something.

I have two 256Mb SD cards which I got from Alec with the Uzebox (along with a third full with games), and I use these two for doing my programming. They are Micro SD's in adapters, only differing slightly in the adapter (the cards themselves look identical). One of them failed for about every second programming in this test. The other didn't fail in 10 tests. Then as an experiment I swapped the cards between adapters. I got no failures with either. Swapping them back I again got no failures.

So it seems like the problem is (or was?) a poor connection within one of the SD card adapters. Since it consistently went away for now (about 70 programmings I guess) it seems like my Uzebox is actually reasonably stable for programming.

But it proves that some kind of verification would be very beneficial! And it also proves that indeed (as CunningFellow mentioned a while back) the SD card could be the culprit, so it could be reasonable to try to catch it before the flash is even accessed (so to not age it with data which is bad anyway). It could also be nice to warn the user about it since the faulty SD will also subtly break games which use the SD for critical stuff (I mean things like Tempest could go with a glitchy SD card, all it would get are a few transient pixels in the webs, but for example it could break an Alter Ego level. Worse if it is something which isn't directly visible to the player).

Anyway, I attached the test program pair (you should program them alternatively to test, they use different patterns so a stray remaining pattern can't falsify the results) so you could try it yourself with your hardware if you suspect programming glitches, and maybe you could find the source with it.

EDIT: Aand...

For a few minutes I thought I succesfully bricked the Uzebox!!! I mean I continued the tests, then with the card (or adapter?) which was problematic first I managed to perform a programming after which it couldn't recover (the bootloader did start, but couldn't program, not even from the good card) until I power cycled it. Then it worked again, the faulty card neither showing any symptom.

[Jubatian]

So I finally tried the programmer as well. A bit mixed results. But the good thing is that the Uzebox still lives.

It wouldn't be so if I didn't apply some forethought. After I managed to figure out how to interact with the programmer using AVRDude and then how to actually connect with the ATMega, the first thing I did was making a binary dump of its flash.

Then I tried Xiaolong's bootloader. Nothing, nada, dark screen, no signal. Back with the binary dump. Uzebox works again, it is possible to program. Then another try, now using the bootloader from the Uzebox master branch. I uploaded the hex file, the result was that the bootloader apparently started (after removing the programmer and then plugging in an SD), the SD card contents showed, but I couldn't carry out any successful programming with it.

So back to the binary dump. It works again. Weird. The bootloader reports the same version number (I didn't do a binary compare yet). There is clearly some difference between Xiaolong's bootloader (no signal at all) and the one from the master branch, but neither works as a bootloader. I don't know what could be amiss. The hex file contains a load address, so I would think it was programmed into the proper region (and if it wasn't I guess instead of Xiaolong's no signal I would have got the normal bootloader since if it was loaded on the bottom for some reason, the original bootloader would have remained intact).

Anyway, how do you program a bootloader? Anything specific to a USBTiny programmer? (If it does matter, my avr-gcc version is 4.8.1)

[xiaolong]

Hi Jubatian, nice you have programmer now and it clearly works, too bad my bootloader had problems. Here is a hex file of bootloader to try, hope better luck with this one. For bootloader compiling and programming I use Atmel Studio 6.2 Otherwise have used AVRDude for programming on both Linux and Windows and it works great

[Artcfox]

So it seems like the problem is (or was?) a poor connection within one of the SD card adapters. Since it consistently went away for now (about 70 programmings I guess) it seems like my Uzebox is actually reasonably stable for programming.

Back when I first wrote my SD/FAT12/16/32 routines I soldered a 0.100" pin header onto a microSD -> SD adapter, so I could plug it directly into my breadboard, and it was then that I realized that most adapters are total crap, and fail after a few dozen insertions and removals. The SanDisk adapters in particular are the most crap, some have broken in half after just 3 insertions and removals. I've had good luck with the Samsung adapters though.

I've thought about just making up a PCB that lets me use a real microSD card socket, and shape the rest of the PCB like an SD card with copper "fingers" that match up to where the fingers on a real SD card are, but I haven't gotten around to that.

[Jubatian]

After several uploads and downloads I verified that I get somehow broken builds of the bootloader on the master branch. During programming it randomly skips pages writing them with 0xFF. So for now it seems I can't test anything derived from the original bootloader. (And still I have only one thing which works: The binary dump I precautiously made before breaking anything)

I will check that source (it isn't very big), possibly there is something in it (C language undefined behaviours?) which causes miscompilings. GCC in my opinion is very stable and dependable, but only so long you used the C language by the standard, so I doubt what I see here were actual compiler bugs.

EDIT: Xiaolong: That bootloader works! At least I could do successful programmings with it (no errors experienced, but given that it was caused by a micro-SD adapter it doesn't prove anything on whether reliability improved in any manner), and it is seemingly even faster than the normal bootloader. It is grayscale, though.

Just a thought for the SD adapter issue. I think I got the nastiest type of failure here. Clearly a MISO line instability. If it was a clock or MOSI, I would have received complete sector faults, the SD reading totally breaking, so a clear failure where it was unlikely that the programming result worked at all. The MISO skipping random bits rarely is totally evil.

I picked up the bootloader code and modified it removing interrupts (probably the same change like Xiaolong did). Compiling and uploading this it works proper! (So first time I got a functional build on my system). Possibly there is some quasi-deterministic race condition in the master branch which is triggered depending on the cycles some stage there takes, which of course varies depending on which C compiler you used.

EDIT: I did some hack-job on the bootloader, replacing the mmc.s source to a new SD card interface routine pack which attempts to verify reads. I did not implement SD CRC checks, rather simply a re-read, I settled with this since without enabling CRC on the commands as well you could also get wrong data by a corrupted read command (fetching a sector from God knows where and its CRC will be correct). The '-' before the files is somehow mangled, it comes from the code on the master branch (I didn't try to discover exactly why it happens for now, it seems to be working). It doesn't check whether the flashing succeed. Interrupts are of course off as compiling the original code (with sei's and cli's littered around in the flashing code) doesn't even work for me, so no progress bar. The HEX file is included in the "default" folder.

EDIT: So continuing the quest here... This is nasty. The make has a flag to set the bootloader to address zero presumably for testing, and I used it. Possibly someone hacked up that Print function some time later in such a manner that this didn't work, so it had to be hacked back. The goal was to get it starting in the emulator, and finally it did. The video signal is ugly, at least it seems so. Then finally I found that weird first character thing. That character is a consequence of reading stuff from the SD card from a sector selected by uninitialized memory! Yuck . At least it is never explicitly initialized, possibly being zero, so you will get whatever is in the MBR at the position where the name of the game should begin.

[uze6666]

Many past threads discussed the idea of putting SD functions ( among other) in the bootloader, like a BIOS. Personnaly, my stance has always been to avoid doing this for one main reason: Games depends on a specific version of another piece of software which you can't really control if it's there or not. Another thing that annoys me is what if we were to find a bug in that shared code for which the fix breaks many games? Having to run around the sources and recompiling them has never been too much palatable to me. Also, like many, I heavily use the ISP flasher during development, so that naturally erases the bootloader. I would otherwise have to flash two separate programs or link the bootloader to the game first.

I much prefer to keep the bootloader separate from the games. Sure it creates some code duplication, but in my eyes, the other option is just so much more complex and brittle. I'd also prefer to keep space for improvements. The CRC being one, but also mainly SDHC support. The visual side could also get a bit of love. Bootloader self-upgrade could be nice, if it's possible. The bootloader area is 8K so in theory it may be possible to put some upgrade code in the lower 4K. Though current Uzeboxes would need to be upgraded with a new bootloader that support self-upgrade.

That said, I agree the bootloader could and should be improved with the checksum idea. Removing the progress bar could be done I guess, but I don't like it much because most LCD TVs loses sync and displays those blue screens just like there is no signal. If we can find a way to disable it via a menu option of some kind, I'd be ok.

Uze, where the kits shipped with fuses that prevented bootloader re-flashing ?

No, I don't recall that I enabled that fuse.

I will check that source (it isn't very big), possibly there is something in it (C language undefined behaviours?) which causes miscompilings. GCC in my opinion is very stable and dependable, but only so long you used the C language by the standard, so I doubt what I see here were actual compiler bugs.

GCC version differences could well explain the problems you have when compiling from the master. The binary dump you have was probably compiled with an old GCC version like 4.7. I use the same HEX for my kits since many years. Perhaps something to do with struct aligments or padding and such.

[Jubatian]

For the loading bar I remember I discovered some nasty race condition there (between sync interrupts and programming), but now I forgot what it was exactly. If I remove the loading bar, I can compile and run the bootloader properly, otherwise it won't succeed programming, which I attributed to this (and that different GCC versions likely generate different code which might or might not trigger the race condition). I didn't look at it since a while, too much involved in other stuff.

The idea of a BIOS, I understand can have such problems you mention. I mainly thought about the SD reading routines there: if they were exposed, a benefit would be that all games using that interface would be able to use the SD card if the bootloader was capable to load the game from it. This could be an improvement (later fixes in the SD code allowing it to support for example more cards or better stability would also improve the games). In overall within the bootloader that's the only chunk of code which would really worth exposing (along with possibly a programming interface). So even if it happened, it wouldn't be for "everyone" (only for games which actually wanted to use the SD card in some not overly complicated manner). This SD area is something which just gets things broken no matter how you approach it: if you let (of "force" by not having an interface at all) games rolling their own SD code, they will be stuck with that, and that would also be extremely hard to fix if it turns out that they wouldn't support lots of cards (since the SD protocol itself is quite brittle, every card doing it a little differently).

For some idea on the thing, look at the comments of the top of this code: https://github.com/Jubatian/cuzebox/blo ... cu_spisd.c , this is CUzeBox's SD card module. I aimed to get rather strict emulation, but it turned out that doing it would just break games, while SD cards by the specs could possibly do those.

[uze6666]

For the loading bar I remember I discovered some nasty race condition there (between sync interrupts and programming), but now I forgot what it was exactly.

If you ever recall let me know, I would rather try and fix that then to remove the bar altogether. One simple thing that could be done is use a flag in the eeprom do activate or deactivate the progress bar.