Downtown Doug Brown

As I mentioned in my last post, I spent a good chunk of my spare time over the past 6 months working on a project I’ve been thinking about for over a decade. I bought a Chumby 8 in 2011. It’s an 8″ touchscreen device powered by the Marvell PXA166 processor. It is essentially a souped-up digital picture frame with extra capabilities like speakers, a microphone, and Wi-Fi. There are a bunch of little Flash-based “apps” you can install for stuff like pictures, music, sports scores, weather, games, etc. I have no idea how many of the apps still work these days. Chumby actually went out of business a few months after I bought mine, although one of the founders stepped up to keep the service running. A variant of this device was also created for Insignia, which was called the Infocast 8″ Internet Media Display.

I was never really interested in using it for any of the stock functionality. I thought it would be a fun development platform. It would be exciting to make some custom apps in Qt or something. One thing that was a little frustrating was that it came with a Linux 2.6.28 “Erotic Pickled Herring” kernel circa Christmas 2008, which was ancient even at the time I bought it. This is a pretty common issue with Linux-based devices. I will even admit I’ve been responsible for some old kernels out in the field in Internet-connected devices. I don’t blame Chumby. It’s tough when the SoC vendor doesn’t submit their kernel modifications upstream or at least keep their fork up to date. I’ve been there.

I didn’t really want to stick with the stock kernel, but there wasn’t much of a choice. There wasn’t a newer kernel available and the mainline kernel didn’t support everything needed for the Chumby. I guess in my mind that limitation was enough to sidetrack the whole project. I didn’t want to play around with kernel 2.6.28. I wanted to play around with modern things. The Chumby went into a cabinet somewhere and I didn’t think about it for a while.

Fast forward a couple of years to 2013. I had gained more experience with Linux kernel development by rolling out the firmware for a new product. Side note: that product sadly used the 2.6.37 kernel; I’m sure someone reading this post will see the 2.6.37 version number and know exactly which vendor that was. Anyway, at the time, Linux 3.13-rc1 had just been released. I soldered on the UART header and tried to get my Chumby booting into that kernel version, and eventually (sort of) succeeded. I was very much a Git newbie at the time, so I didn’t really know what I was doing in terms of collaboration, but I shared some patches based on 3.13-rc3. Fellow Chumby forum user dusk brought them into a branch and did some extra work to get some additional pieces working.

Then, life got in the way and I didn’t touch the project again for over 8 years.

Looking back on everything now, my work on the Chumby kernel in 2013 was kind of poorly timed. The ARM kernel was in the process of being migrated over to use device tree, but my patches weren’t using the device tree. I should have been trying to get everything working using device tree, but I didn’t have any experience with it at the time. Now in 2022, I have a lot more experience with Linux kernel development. I know much more about using device tree than I did in 2013. I hadn’t yet developed all the skills necessary for this project back then.

With that initial explanation out of the way, in 2022 I’ve been getting my Chumby working with the mainline Linux kernel and slowly trying to submit fixes upstream for issues as I find them. To be clear, I’m not trying to get the stock Chumby software working with the new kernel. That’s likely impossible. The stock software is heavily dependent on Flash which is a dead end. I’m just getting the new kernel running well enough so that I can develop my own custom software for it. I’m going to write a few posts about that process and some of the fun challenges I had to overcome. This first post in the series will talk about some of the work I did with U-Boot.

Earlier this year when I started working on this again, I set a ground rule. I wanted to be able to build the whole bootable SD card image for my Chumby from scratch using buildroot. Previously, I was manually hacking up the stock SD card image’s U-Boot binary (version 2009.07) to change the default environment to load my newer kernel. I wanted something that I could build from scratch (as much as possible). This would be easier to maintain going forward, lead to a better understanding of all of the pieces being used, and as an added bonus, would be easier to share with other people.

Going forward with that requirement, I started out by trying to figure out the boot process. I wanted to use a modern U-Boot under the assumption that it would be more likely to work properly with a modern kernel. For example, one of my hacky patches for 3.13-rc3 linked above was working around an issue that had already been taken care of by newer U-Boot versions.

Marvell products haven’t historically had public technical documentation available, but the PXA16x is an exception. They did actually release a lot of documentation publicly for this SoC. By the way, I’m going to refer to it as the PXA168 from now on. The PXA166 is basically the same thing. After some initial research in the Boot ROM manual, it became apparent that I should try to reuse Chumby’s original first stage bootloader rather than roll my own solution. It appeared to be some kind of proprietary Marvell bootloader with some Chumby-specific modifications. The Chumby folks knew exactly what they were doing with things like PLL configuration, DRAM configuration, etc. and I didn’t want to get too far in the weeds figuring that all out, so I decided to stick with Chumby’s stock bootloader (obm.bin) and figure out how to make my SD card image compatible with it. I confirmed that the linked obm.bin 100% matched what was bundled on my Chumby’s stock SD card at the very beginning of partition 1, which is located at an offset of 0x800 from the start of the card.

I looked at my Ghidra disassembly of obm.bin, a dump of my original SD card image, as well as aspenix-1.0.tgz on Chumby’s source code download page. Between the three of them, I came to the conclusion that obm.bin will load 256 KB of data from the SD card starting at offset 0x10000 of partition 1, store it in the DDR RAM at 0x00F00000, and jump to it. This is the mechanism it uses to load and run U-Boot from the SD card.

This was easy enough to instruct buildroot to assemble through the use of genimage, which is also how buildroot creates bootable SD images for other systems such as Raspberry Pi. Without adding U-Boot to the SD card, I was at least able to confirm that obm.bin was running because I got the normal Chumby bootloader output:

Total times UART was inited: 0x00000001
Total wait loops iterated: 0x000022B3
Hello world, I'm the most incredibly annoying boot process you'll ever meet!
key waiting
done with key wait
key waiting2
flash boot attempt
loading OS loader
chumby!
disableIRQ
setupxfer
levelling up...

With the first stage bootloader figured out, it was time to create a modern U-Boot for the Chumby. Some quick research indicated that U-Boot had actually removed support for the PXA168 (a.k.a. Armada 100) in 2021 because the last remaining board that used it (aspenite) hadn’t been converted to use CONFIG_DM (driver model) by a deadline. Because of that, I started from U-Boot v2021.10, which was the last version that still supported the PXA168.

The nice thing about this project is I have access to an older U-Boot and kernel that are known to work properly, so I can see any custom logic that Chumby had to implement for their board. It’s also helpful to see the schematic. I was able to see that I needed to drive GPIO109 low in order to allow the UART RX signal to pass through a level shifter.

I started with the existing aspenite board config in U-Boot and simply copied it. The only thing I changed was setting CONFIG_SYS_TEXT_BASE=0xF00000 and CONFIG_NR_DRAM_BANKS=1 to match Chumby’s U-Boot configuration (and my understanding of where obm.bin loads U-Boot). Then, I added (ugly, temporary) code in board_early_init_f to drive GPIO109 low as described above.

*(unsigned int *)0xd4019154 = (1 << (109-96));

That’s super hacky, by the way; later on, I cleaned everything up and did it through device tree with a GPIO hog instead.

With those small Chumby-specific tweaks implemented, I built U-Boot and set up genimage to put it onto the SD card at the correct offset. It didn’t work. After banging my head against the wall for a while, I figured out that the aspenite board and the Chumby board have their UART RX and TX pins backwards. On Aspenite, GPIO107 is RXD and GPIO108 is TXD. On Chumby, it’s the other way around: GPIO107 is TXD and GPIO108 is RXD. So I tweaked those pin function assignments in the board file, and it worked! Well, kind of.

U-Boot 2021.10 (May 03 2022 - 15:22:18 -0700)
Chumby 8

SoC:   Armada 88AP168-B0
DRAM:  128 MiB

It printed that out, and then just hung. This thing does have an (unpopulated) JTAG header on it, but I’m not set up to use it, so I used the tried and true debugging method that has always worked for me instead: debug print statements. Clearly the UART was working, so I could print messages during the boot process. I littered them throughout the code. What I found was that it was getting all the way through the initial setup in board_init_f and then failing at the call to relocate_code. At least on ARM, U-Boot is told where it is initially loaded in RAM through CONFIG_SYS_TEXT_BASE, and then as part of its loading process, it eventually copies itself to a different location in RAM and jumps to the new location instead. It seemed to be hanging up when it attempted to jump to the relocated code.

Further diagnostics, which required a little bit of ASM hackery to tell me the current program counter value, led me to discover that pre-relocated U-Boot wasn’t actually executing from the location where I thought it was! I told it that it was running from 0xF00000, which is what all my research into obm.bin had implied. But it wasn’t! It was running from an offset of 0. I’m still not 100% sure what’s going on here because U-Boot definitely also gets loaded into 0xF00000. But it appears that obm.bin also copies U-Boot to 0x0 (or maybe enables some kind of address mirroring). Either way, 0x0 is the actual address it jumps to. Not 0xF00000. I probably could have figured this out by spending more time disassembling obm.bin, but I just wanted to get my U-Boot working.

I decided to try fixing this in a weird way: I hacked the reset code in arch/arm/cpu/arm926ejs/start.S. If the reset code determined it was running from an address less than 0xF00000, it jumped to 0xF00000 instead. This totally worked! U-Boot fully booted after I made that change.

That workaround was silly though. The real solution is simply to change CONFIG_SYS_TEXT_BASE to 0 in order to match the actual pre-relocation base address it’s running from. I changed it and removed my hacky fix. I was so confident that the problem would be solved at this point, but it didn’t work! U-Boot froze. After a good night’s sleep, I discovered that CONFIG_SYS_INIT_SP_ADDR was set to (CONFIG_SYS_TEXT_BASE – 0x00200000) which is not a valid RAM address when CONFIG_SYS_TEXT_BASE is set to 0. If your stack pointer doesn’t point to RAM, you’re gonna have a bad time. So I simply changed it to 0x00D00000 instead. And with that change implemented, U-Boot was up and running! Yay!

U-Boot 2021.10 (May 04 2022 - 09:07:35 -0700)
Chumby 8

SoC:   Armada 88AP168-B0
DRAM:  128 MiB
Loading Environment from nowhere... OK
In:    serial@d4017000
Out:   serial@d4017000
Err:   serial@d4017000
=>

I couldn’t get too excited though. It was up and running, but it didn’t have any support for the SD card, which would be needed in order to load the kernel and boot it. I ported over a bunch of pin function selections from the original U-Boot so that all of the pins would be configured properly for their peripherals. I also went ahead and converted the board to use driver model, which basically just entailed copying over PXA168 device tree info from the Linux kernel source, making a basic Chumby device tree source file, and changing a few options in Kconfig and the defconfig file for the board.

Getting the SD card working actually wasn’t too bad at all. U-Boot already had a Marvell SDHC controller driver (mv_sdhci) and it was already set up for device tree support. I just needed to add it to my existing device tree. I did run into a compatibility problem though. If I tried to read more than one block at a time from the card (which happens when you mount a filesystem) or erase a block, it would fail:

sdhci_send_command: Timeout for status update!
mmc fail to send stop cmd
 ** fs_devread read error - block
Failed to mount ext2 filesystem...

MMC erase: dev # 0, block # 4, count 1 ... sdhci_send_command: Timeout for status update!
mmc erase failed
0 blocks erased: ERROR

After a bunch of tracing (CONFIG_MMC_TRACE was very useful!) I decided that everything was working properly and the error wasn’t really an error. In fact, looking at U-Boot’s SDHCI driver, there was even code that could inhibit this error if the SDHCI_QUIRK_BROKEN_R1B quirk was enabled. As soon as I enabled that quirk, everything started working properly and I was able to read and write the SD card with no problems. Well, there was one problem. Commands would always wait for 1 second before completing, which is something I have also seen in U-Boot on the Xilinx Zynq in the past. I decided to investigate that situation further and found some back and forth where one patch had fixed this delay issue but then it got reverted because it caused issues on another board. I settled on something halfway in between and it seems to work okay for me on this board. I don’t know for sure if any of what I did is the correct fix, but I haven’t run into any SD card issues since I implemented it.

This is starting to get too long to go into detail about everything else I did in U-Boot, but the things I described above were the main challenges. As soon as I had SDHC support working, it became much easier to make further U-Boot progress because I could send new changes through minicom using YMODEM and write them to the SD card, rather than constantly moving the SD card back and forth between the Chumby and my USB card reader. As a quick summary, I converted the mvgpio driver to use driver model, got communication with the Chumby’s STM32 coprocessor working to enable the reset and poweroff commands, added support for displaying an image on the LCD, and ported Chumby’s original U-Boot code for reconfiguring the PLLs and DRAM to work around erratas. The errata workaround implemented by Chumby was crazy and impressive. It required setting up the L2 cache to be used as SRAM, copying code into the SRAM, jumping to that code in SRAM, doing the reconfiguration, and then jumping back into the DRAM after everything was complete.

That’s a pretty decent summary of my U-Boot changes. This work left me with a great foundation to build on for all of my new kernel experimentation. It also gave me a good preview of the kinds of problems I would be dealing with in the kernel. I haven’t done a great job of cleaning up my commits and I don’t have any plans to upstream these changes. Since PXA168 support was removed from U-Boot in 2021, I don’t think it would be worth the effort to try to upstream them. I’m more focused on upstreaming the kernel changes, which I will begin talking about in my next post. In future posts, I will likely come back and mention a few changes here and there that involved U-Boot as I brought up additional peripherals in the kernel.

By the way, all of the work I’ve done on this project also applies to the Insignia Infocast I mentioned above. The biggest hardware difference between the Infocast and the Chumby 8 I’ve noticed so far is that the Infocast uses a revision A0 CPU, whereas the Chumby uses a newer B0 CPU. The A0 has a known silicon bug that prevents booting directly from an SD card, so the Infocast has an SPI flash chip containing a slightly tweaked obm.bin. The SPI flash is unpopulated on the Chumby because the B0 revision fixed the problem and it can load obm.bin directly from the SD card instead. This is all a long way of saying that it was a good decision to stick with Chumby’s stock first stage bootloader. All of the development I’ve done so far works perfectly fine on the Infocast as well.

Click here to go to the next post in this series, where I get the kernel booting with SD card support.