Downtown Doug Brown

I’m going to start this post off with the obligatory list of links to the previous parts in the series if you’re new here and are interested in seeing the full story: 1, 2, 3, 4, 5, 6, 7, 8, and 9. This is the tale of how I upgraded my Chumby 8 to run a modern Linux kernel.

I only had some minor things on my list to figure out before I could call my kernel upgrade good enough to be finished. The most important remaining thing was the real-time clock, or RTC. I noticed that whenever I rebooted the Chumby, it always started out with the date set to the Unix epoch in 1970:

# date
Thu Jan  1 00:00:12 UTC 1970

I had observed earlier that the default pxa168.dtsi file in the kernel had an RTC already added, but it was disabled:

rtc: rtc@d4010000 {
	compatible = "mrvl,mmp-rtc";
	reg = <0xd4010000 0x1000>;
	interrupts = <5>, <6>;
	interrupt-names = "rtc 1Hz", "rtc alarm";
	clocks = <&soc_clocks PXA168_CLK_RTC>;
	resets = <&soc_clocks PXA168_CLK_RTC>;
	status = "disabled";
};

Would it really be this simple? Did I just need to enable the RTC in my device tree file, make sure the kernel driver was enabled, and then be done with it? I had already experienced similar success with other PXA168 peripherals. So I tried it out.

&rtc {
	status = "okay";
};

I also had to set CONFIG_RTC_DRV_SA1100=y. Why that driver? I don’t see SA1100 mentioned anywhere in the blurb above. Well, it’s because that’s the driver that contains the device tree compatible string for mrvl,mmp-rtc:

#ifdef CONFIG_OF
static const struct of_device_id sa1100_rtc_dt_ids[] = {
        { .compatible = "mrvl,sa1100-rtc", },
        { .compatible = "mrvl,mmp-rtc", },
        {}
};
MODULE_DEVICE_TABLE(of, sa1100_rtc_dt_ids);
#endif

The Kconfig file mentions a few PXAxxx variants in conjunction with this driver. It doesn’t specifically mention the PXA168, but it’s the exact same peripheral.

Anyway, with these changes made, I made sure a good battery was installed, and then powered my Chumby up using the new kernel. During the boot process, an RTC was detected which also seemed to have its date set to the Unix epoch, and it set the system clock:

sa1100-rtc d4010000.rtc: registered as rtc0
sa1100-rtc d4010000.rtc: setting system clock to 1970-01-01T00:00:00 UTC (0)

This was fairly promising, right? I already had a good amount of experience dealing with RTCs in Linux, so I understood what would be involved in testing them. First of all, the message “setting system clock to…” above meant it was reading that 1970-01-01 date from the RTC. That’s because I had CONFIG_RTC_HCTOSYS=y. If you have your kernel configured that way, it’ll automatically attempt to set the system date and time from the RTC date and time at startup. Clearly the RTC didn’t have a good date written in it, which makes sense because I had just put in a good battery.

I wrote a random date into the RTC for testing. First I set the date on my RTC, and then I used hwclock to write it:

# date -s '2022-09-22'
# hwclock -w

Then I read back the RTC to see if all was good:

# hwclock
Thu Sep 22 00:00:41 2022  0.000000 seconds

Hooray! The RTC driver definitely seemed to be working. It was writing a date into the RTC, and then after writing it, it was able to read it back out. It was even advancing properly — I had read back the date 41 seconds after writing it in.

“That was easy,” I confidently said to myself as I rebooted once more to see if the changes stuck. I brushed off my shoulders, proud of a job well done. I already knew it was going to work perfectly. Testing it was just going to be a formality. The following text popped up on the screen as the kernel came back up:

sa1100-rtc d4010000.rtc: registered as rtc0
sa1100-rtc d4010000.rtc: setting system clock to 1970-01-01T00:00:00 UTC (0)

What the heck? I had successfully written to the RTC, but it was right back to thinking it was 1970. Not that I had any reason to doubt the console text I saw above, but using hwclock I confirmed that the date in the RTC had definitely reset:

# hwclock
Thu Jan  1 00:00:55 1970  0.000000 seconds

I double- and triple-checked that the battery I had inserted was good. Basically the RTC was working fine when the system was powered, but even the act of rebooting was enough to make it forget the date. I would expect that an RTC should also remember the date/time when the system is powered off. Obviously that wasn’t going to work if a reboot didn’t even preserve the date. So much for testing just being a formality. That’s what I get for being overconfident! Back to the schematics I went.

When I looked at the schematics, I realized I should have started there. I should have followed my own advice that I had followed on other parts of this series like the audio support. I had made a terrible assumption. My assumption had been that the Chumby 8 actually used the PXA168’s internal RTC. That turned out to be very incorrect.

I can’t thank Chumby enough for making the schematics for the Chumby 8 available to the public. It really saved my bacon on this project. I’m sure I could have figured out a lot of this stuff without it, but having the schematics readily available really saved me a lot of time.

It turns out that the coin cell battery on the Chumby 8 doesn’t go to the main processor at all. In fact, further reading about the PXA16x’s built-in RTC seems to reveal that it’s basically useless. The Armada 16x software manual points out that in hibernate mode, it stops counting (page 220). If that’s the case, I would also expect it to stop counting when it’s powered off. I don’t see any pins labeled VRTC or VBAT in Marvell’s pinout, which I would expect to see for hooking up a battery. I did see a note in the hardware manual‘s version history (page 16) saying “Added note that the RTC has no external hardware connection” though. That’s a funny way of saying “the RTC built into this SoC is useless.”

What Chumby did instead was use the battery to power the STM32F101 “cryptoprocessor” that I’ve mentioned in the past:

This was the circuit I had expected to see going to the PXA168. But thinking about it further, it makes a ton of sense. This solution is so much more flexible than a conventional RTC. The STM32 always being on would have allowed Chumby to implement all kinds of other fun stuff like automatic scheduled powerups. I guess a lot of normal RTCs have alarms that can do stuff like that too, but it seems like you end up with ultimate flexibility by going this route. Either way, it appears Chumby had no choice anyway, since the PXA168’s RTC doesn’t have a battery power input. If I’m wrong about this, I’d love to be corrected!

I had slowly gained more familiarity with the Chumby’s boot process during this project, so I started looking into the original boot scripts to see if I could find something about setting the date. I knew that the original Chumby firmware would remember the date between power cycles, so I just had to figure out where it was handled. I didn’t see anything related to the RTC in the actual kernel source, so I suspected it was handled in userspace.

I went back to a backup of my original SD card containing Chumby’s stock firmware. There are two rootfs partitions and then a third partition which ends up being mounted as /mnt/storage. I looked through the boot scripts in /etc/init.d. rcS.background in particular contained a lot of interesting info, including this relevant chunk:

# Attempt to restore time based on the crypto processor's uptime counter
echo "Attempting to restore time from CP"
message_start "Restore time from CP"
restore_time
echo "Date is $(date)"
message_end "$(date)"

This comment revealed some interesting info. It confirmed that the STM32 is responsible for remembering the date. It also told me that I would be looking for an uptime counter. So the STM32 wouldn’t be directly tracking the date and time; it would just be keeping track of how long it had been powered on. Remember that the battery keeps it powered even when the Chumby itself is off.

I found restore_time as a perl script in /usr/chumby/scripts. What it does is actually pretty simple. It reads the uptime (in seconds) from the cryptoprocessor using /usr/chumby/scripts/cpi.sh, and also reads a couple of local files: /psp/cp_offset and /psp/cp_time. cp_offset contains the time that the cryptoprocessor booted up as a Unix epoch. cp_time contains the uptime in seconds that the cryptoprocessor reported the last time we checked.

If the cryptoprocessor’s current uptime is less than the content of cp_time, then that means it has lost power since the last time we looked at it, and thus we don’t know what time it is. Otherwise, it adds cp_offset to the newly determined uptime, and that is the actual time it sets the system to using the “date” command.

cpi.sh is simply a wrapper that calls the /usr/bin/cpi utility and retries if it fails for whatever reason. It talks to the STM32 through a UART, using the same communication channel I discussed in part 4 when I got poweroff and reboot working. The UART is totally free during normal operation so that a userspace utility like cpi can use it. The reboot and poweroff functions will take over control of the UART only when they are invoked.

I also found a perl script next to restore_time called save_time. It writes out the cp_offset and cp_time files. It’s called by /usr/chumby/scripts/sync_time.sh, which is a script that first syncs up with an NTP server to get the actual date and time, and then runs save_time so it’ll be ready if the Chumby is later powered off or rebooted. This sync_time.sh script is called in a few different places, including when the network comes up.

So that’s how the Chumby manages its “RTC” if you want to call it that. It’s basically just a software RTC emulated by keeping track of the uptime of the STM32. Here’s a block diagram of the setup.

I actually like this solution a lot! It doesn’t require any special kernel logic. It’s entirely implemented in userspace. No special kernel drivers are involved whatsoever. This is a concept that I’ve come to appreciate as I’ve gained more Linux experience through the years. If you can solve a problem in userspace, why bother force-fitting it into the kernel? Chumby probably could have figured out a way to make the STM32 show up as an actual /dev/rtc0 device…but what would the advantage have been over the scripts they wrote? None. It would have been needless overcomplication. Overall, I think the userspace solution they settled on is excellent.

How could I make use of this with my modern setup? I knew I could easily make scripts just like the stock Chumby ones assuming I could somehow access the cryptoprocessor’s uptime counter. Luckily, I found that Chumby’s cpi utility that originally ran in Linux 2.6.28 using a much older glibc still worked fine in my newer environment. I even found the source code and binaries for it on GitHub, so if the binary hadn’t worked for whatever reason, I likely could have rebuilt it instead. The GitHub cpi binary wasn’t exactly the same as the original cpi binary I had found on my Chumby’s SD card, but out of convenience I opted to use it. It worked just as well. I added it as a package in my custom buildroot. Then I wrote some scripts for saving/restoring the date and time. They were very similar to what I found in my Chumby’s rootfs, but they were written as shell scripts instead of perl.

After I enabled NTP to keep the time synced up using time servers on the internet, I wrote a simple daemon that automatically restores the date and time at startup, saves it after NTP has successfully synced up, and then re-saves every 24 hours from that point on just to make sure everything is happy.

Originally I was saving these files directly to the rootfs but I eventually opted to make a separate settings partition on the SD card to use for storing settings like this instead. I like to keep the rootfs read-only if I can.

With the solution I described above fully implemented, now when I boot, I can immediately log in and run the date command, and it looks perfect:

Welcome to Buildroot
buildroot login: root
# date
Sun Jun  2 16:35:01 UTC 2024

Overall, the whole process of getting the “RTC” working properly went pretty well! When I originally discovered the PXA16x’s built-in peripheral wasn’t going to work, the first thing that popped into my head was “oh no, here we go again” but the solution still ended up being pretty simple. Kudos to the original Chumby developers for coming up with a nice solution for saving the date and time. I didn’t have to jump down into any crazy kernel rabbit holes this time.

This project is definitely starting to wind down now, but there’s still more to come. In the next post, I want to talk a little bit about the SD/CF/MS card reader that’s built into the Chumby and how I got it working as closely as possible to the original firmware’s approach.

Click here to go to the next post.