Frequent programming defects - bootloader problem?

[Jubatian]

I often check my game design on the "real thing" to see how they look and feel like. However I frequently run in the problem that uploading the game introduces small defects. Once I got a case which was very clear as the data was corrupted, a sprite tile having a byte zeroed (which on Mode 74 with normal blitting appears as two adjacent transparent pixels). Other times I occasionally get odd but consistent behaviours (that is, consistent until I reupload), likely for the same kind of flaw occurring elsewhere in the binary.

I confirmed that it is not the SD card (at least not the data on the SD card). Reuploading the game directly is normally seemingly impossible (as the bootloader refuses to do it if it detects that the ROM already contains it, didn't check what it seeks for to determine this), but by uploading another game, then trying again it is possible, and that makes the odd behaviour going away.

This is not a show-stopper, but something quite annoying. I think about a third of the uploads (assuming a ~60K game) miscarries by how frequent I get to see obvious defects (about half the case I believe it goes unnoticed).

My first thought is that overclocking is the source of the problem (assuming that the uploading takes place at the same 28MHz clock, which I think is the case since it has a text progress bar which would probably be impossible otherwise). My Uzebox doesn't show any other odd behaviour.

Wouldn't it be possible to add a verify step to the upload to check (by comparing to the SD data) whether the upload succeed? And - or dropping the upload progress bar to rather upload at a lower frequency? (such as 14 MHz) It doesn't take too long, and the progress may still be indicated by flashing a LED.

By the way, just out of curiosity for now: is it possible to replace the bootloader without a programmer? (Of course I see that even if so, there is a good chance of bricking the Uzebox... Well, I have access to a programmer at my workplace if I screwed up, not like I want it anytime soon, just a sudden impulse of curiosity)

[CunningFellow]

I have only had uploads from SD card fail maybe 1 in 100 times.

However ArtcFox said he regularly had these problems and also was asking about modifying the bootloader with a verify step.

I think it sounds like a good idea myself. However you probably also need to add CRC checking to the data read from the SD card because there could be two points of failure for a corrupt upload.

SD_Card to RAM
RAM to flash

It probably needs a bit work to get new features squeezed into the 4K boot block, but I think we are in a good position with a few ASM programmers on hand to get that done.

You can only have the AVR flash its own bootloader area if the fuse bits have been set to allow it.

Edit: Any reason we can't support SDHC while at it?

[Jubatian]

There is no great need for CRC, and it won't really fit. A proper Verify would check both steps (since it re-reads the game from the SD card as it doesn't even fit in RAM). The CRC would either need a complex algorithm or a table (512 bytes for a 16 bit CRC), the latter certainly wouldn't fit.

If you brought up SD support... Well, the bootloader needs SD access. It bugs me that why it wasn't exposed as an interface, that way anything needing SD access wouldn't have to lug around extra code for that. The bootloader may even support fragmentation since there it doesn't matter that much (it only makes uploading a bit slower to seek for fragments between 512b blocks), that way only specific games would demand an unfragmented SD card (and those which don't stream would likely be all fine with what the bootloader's interface provides, that also inherently allows for later refining of SD code such as introducing SDHC support).

[D3thAdd3r]

It bugs me that why it wasn't exposed as an interface, that way anything needing SD access wouldn't have to lug around extra code for that.

That always bothered me. Even as you guys are finding ever more clever ways to squeeze out more from the resources, a normal simple non-Tornado 2000-ish SD game throws away kilobytes of flash on SD access with PFF. Is there some way, knowing the address of the functions, that we can abuse the existing bootloader to access SD or would it need to be rewritten with that in mind since the ram layouts are different and can't be bent? Uzebox BIOS at least for that would be nice.

The immediate problems might be the fact that the majority of standard kit Uzeboxen have the stock bootloader and the owner likely doesn't have an ISP since he never needed it. Also, the emulator would need to support it before it would be adoptable. Simple SD makes the issue more bearable at least.

@Jubatian - are you using the "cheap ass 256m card" that comes with the kit? Maybe the issues lies in the analog domain and there is a cold solder somewhere messing with the wave shape at that speed. Otherwise I wonder if slowing the SPI clock down, or longer delays, or on bigger cards using partitions could help. Cunning found some things while doing the Uzebox in a controller design I recall and Harty and others have done such things with some positive results:http://uzebox.org/forums/viewtopic.php? ... der#p13516,http://uzebox.org/forums/viewtopic.php? ... 946#p10599. Personally if the bootloader can do SDHC or fragmentation and fit, why not. I don't see it as critical since a lot of SD games wont support it for performance reasons. Certainly some sort of verification would be a more important feature.

[CunningFellow]

There is no great need for CRC, and it won't really fit. A proper Verify would check both steps (since it re-reads the game from the SD card as it doesn't even fit in RAM). The CRC would either need a complex algorithm or a table (512 bytes for a 16 bit CRC), the latter certainly wouldn't fit.

CRC16 will only take up a few 10's of words in flash done tightly in ASM. Would save a 2nd read of the SD Card that is slow. Just

Read to RAM
Check CRC
Flash from RAM
Verify Flash against RAM

If you brought up SD support... Well, the bootloader needs SD access. It bugs me that why it wasn't exposed as an interface, that way anything needing SD access wouldn't have to lug around extra code for that. The bootloader may even support fragmentation since there it doesn't matter that much (it only makes uploading a bit slower to seek for fragments between 512b blocks), that way only specific games would demand an unfragmented SD card (and those which don't stream would likely be all fine with what the bootloader's interface provides, that also inherently allows for later refining of SD code such as introducing SDHC support).

What else could be built in to save space? Button input routines ? That would also make a nicer hardware agnostic method there. The NES/PSX/Atari/Jama button-ness would be programmed into the bootloader.

EEPROM Read/Write, WaitUS ?

Is there some way, knowing the address of the functions, that we can abuse the existing bootloader to access SD or would it need to be rewritten

You would expose the functions via a jump table located at TOP-(n*2). That way the address the application calls never changes even if a newer version of the bootloader changes where the real function lives.

The immediate problems might be the fact that the majority of standard kit Uzeboxen have the stock bootloader and the owner likely doesn't have an ISP since he never needed it. Also, the emulator would need to support it before it would be adoptable. Simple SD makes the issue more bearable at least.

Yes - that is a bigger problem. People not being able to play new games because they have the old bootloader.

Uze, where the kits shipped with fuses that prevented bootloader re-flashing ?

4K is pretty tight already. I'm up for the challenge of trying to squeeze it all into 4K if people can come up with what they think the API calls should be.

[Jubatian]

Drat... I see where my idea ran ashore.

If the fuses are (were) set so, the first important thing would be devising a safe method for updating the bootloader! That is, so hopefully very few Uzeboxes would bite the dust in the process for any accident.

A desperate possibility which only works if there is only one bootloader in existence is mapping it, determining where the entry points are, what signatures they use, what regions of the memory they write and read. Then create the new bootloader with these entry points and memory usages fixed according to the old. Of course this would be a rather limited approach...

CunningFellow: That SD access isn't so terribly slow like you think. It is just 60 kilobytes. In one frame (1/60th sec) if you had no inter-sector gaps, you would read about 25K, and those inter-sector gaps are neither terrible on this scale (if it completed within some 200 milliseconds, the user would never notice it being sligthly slower). The CRC in general might be useful if implementing the SD card CRC (here its polynomial is listed), so it can be included in the interface (probably it is possible to to create code which calculates it on the fly alongside reading the SD data).

D3thAdd3r: Yes, I use those rinky-dinky SD cards. They however appeared to function proper when I was doing SD access stuff like they should (they are not overclocked in any way, the SPI runs at 14MHz, which is way below the 25MHz they should support), probably only having long delays (I didn't ever measure it, just saw they work when properly waiting for the sectors). I assume the bootloader used them right (and if it didn't wait proper, it would show different symptomps than I experienced, certainly it wouldn't produce a random zero byte for that kind of flaw). I will try other cards some time, but again, it seems to be something beyond reach here (just the same as always, no shop having the stuff I need... I am going to tear hair out if I find a single 2Gb card in town for the equivalent of some twenty dollars).

I don't think the emulator support would be something too complicated. Just include the bootloader binary and copy it into the appropriate area before loading the game (so games which overwrite it would still work).

[CunningFellow]

Back of napkin calculations.

I think I can fit bootloader, SDHC support, PetiteFatFS (equivalent), Hardware agnostic Button Support, EEProm Read/Write and WaituS plus a jump (BIOS entry point) table into 4K.

That is assuming there is no major over riding reason for SDHC to not work on some old hardware.

Is it worth it?

Still need to hear from Uze/Harty/Clay if the hardware they sold is fuse protected.

[CunningFellow]

A case for CRC.

I think CRC is the most sensible thing.

If you just test for errors by doing a verify read. If you get a miss-match - How do you tell if it was

First SD Read
Write to Flash
Second SD Read

That was at fault.

I've already done CRC on read in C for ArtcFox to check his hardware (http://uzebox.org/forums/viewtopic.php? ... 120#p16206).

It should not be difficult to turn that into ASM code.

[Artcfox]

I still have flashing problems, and I often have to reflash a game many times before it works, and even then I'm not sure that it flashed completely correctly.

I'd love to see CRC verification added to the bootloader.

[Jubatian]

If you get a miss-match - How do you tell if it was...

It doesn't really matter: it failed. As far as I know from the stuff we have at the place I work, all programmers use the write and verify method (from old UV EPROM burners to PIC & AVR stuff). If either step fails, the programming just failed. Try again. The point is that it is very unlikely that the read stage would systematically fail at the same bit the same way (for the programming and the verify steps), so the method is reliable. The nice thing is that this method doesn't need any change in the source format (compatibility). Otherwise using the SD card's sector CRC is also fine since it can check both the correctness of the read and the correctness of the programming (of that given sector), also without the need for any change in the source format. This method is also very reliable, notably a 16 bit CRC can detect 16 bit burst errors, so if the case is really bytes occasionally turning zero (as one occasion for me suggested it), those would be detected at 100% certainity. Adding a global CRC I think is unnecessary: it feels very unlikely (at least for the purpose of gaming, this is not a system controlling an airplane) that the PC would fail when writing to the SD card, then from that point the sector CRCs are there to verify the data.

The fastest hack solution I think would probably be adding the SD CRC check if it fits. I guess the programming already works sector by sector, to the read stage the calculation of the CRC would have to be added, then another new small routine to calculate the CRC of what was programmed (so reading it back from the ROM). If the comparison between the two fails, either the read or programming of the sector failed. Revert to try the sector again (maybe limit this to one occasion, so in case of some software bug, the thing wouldn't destroy the microcontroller). Read and verify would demand a second read of the SD sector for a base of comparison, but otherwise it is about the same (the second read may make it more complicated, needing different SD commands).