Aug
03

Upgrading my Chumby 8 kernel part 13: the end

Doug Brown Chumby 8 kernel, Linux 2024-08-03

This is the final post in my blog series about getting my Chumby 8 from 2011 working on a modern 6.x Linux kernel, as opposed to the stock 2.6.28 kernel it came with. Here are links to all of the previous articles where I went into detail about the various obstacles I had to overcome: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12. I guess we’ll finish it off with lucky number 13.

It has been over two years since that fateful day when I started trying to get a newer U-Boot working on my Chumby to kick off this whole process! I haven’t been working nonstop on the Chumby this whole time though. I completed most of my work on this project by early 2023, and since then I’ve just been trying to catch up on writing these posts and also fixing little issues as they’ve come up. It’ll probably never be fully completed though. I will likely run into various problems in the future as new kernel versions come out and things inevitably break.

I’m not going to cover U-Boot as part of this. If you’re curious about that, check out the first post in the series. Most of the work I did was in Linux. Let’s start by looking at everything I accomplished in the kernel, grouped by subsystem. We’ll see the commits that are in the mainline kernel versus the commits that I haven’t submitted upstream for whatever reason. I’m also linking to all of the original patches on lore.kernel.org in case you want to see the related code review and discussion.

clk

Mainlined commits (series V1):

Commits only in my branch:

I might have gone overboard making a bunch of tiny commits in the stuff that I submitted upstream, but basically I added several missing clocks and fixed a few mistakes. This was needed pretty early on because I couldn’t set up the peripherals if I couldn’t hook up their clocks in the device tree. The SDHCI controller clocks in particular were a pain to deal with because each pair of controllers has some shared bits that weren’t represented correctly in the existing code at all.

The SSP and TWSI commits haven’t been upstreamed purely out of laziness. They are both good fixes and would likely be easily accepted. The SSP one needs to have its title fixed though. The GC300 and PWM peripherals are both a bit weird to turn on, and I’m currently handling their clock enablement in U-Boot. I’d probably need to come up with a cleaner way to handle them in Linux if I submitted them upstream.

etnaviv

Mainlined commits (series V1, V2, V3):

This was all I needed to do in the kernel in order to get the GC300 2D GPU working. It simply entailed porting over some quirks found in the original Vivante driver which hadn’t made their way to the open-source etnaviv driver. Of course, there was a bunch of userspace work involved too.

ASoC

Mainlined commits (series V1, V2):

Commits only in my branch:

If you’re not familiar with ASoC, it stands for ALSA System on Chip and has to do with audio support. The mainlined commits added support for the WM8961 codec to be used with device tree. I was worried that if I didn’t at least submit them upstream, the driver would eventually be stripped out because nothing was still using it.

All of the custom stuff still in my branch is the work I did to get the pxa-ssp audio driver working on the PXA16x. It started out with fixing a null pointer access bug, eventually involved porting over some unfinished work I found on old mailing list posts for PXA16x compatibility, and then concluded with a bunch of tweak commits to make it fully work.

Most of these commits would require significant cleanup before I would ever consider submitting them upstream. Some of the later commits are fixing mistakes I made in the earlier commits. I’m unsure if my fixes actually work correctly on other PXA architectures too. The “switch to network mode” commit is also interesting because it fixed a weird problem I ran into that caused random static noise to be played in certain situations after I had used the microphone, but this driver was intentionally switched away from using network mode over a decade ago. I didn’t particularly want to go any further down that rabbit hole, so these commits will likely stay in my branch forever.

dma

Mainlined commits (patch V1):

Amazingly enough, this is the only change I made to DMA stuff. The DMA just worked out of the box. I didn’t really need to make this change; it just inhibited a bunch of annoying warnings during the boot process.

I did make a silly mistake in this patch and forgot to add a closing “)” at the end of my Fixes: tag, which made it through to linux-next before being spotted. Luckily the dmaengine maintainer fixed it for me!

ARM

Mainlined commits:

Commits only in my branch:

The two mainlined commits fix a couple of showstopper bugs. The first one made it so the kernel wouldn’t hang when it disabled unused clocks. The second one fixed an issue that caused top to erroneously report 100% CPU usage.

All of the other commits are just inserting peripherals into the device tree include file. Most of them could definitely be submitted upstream to fill out pxa168.dtsi and demonstrate how many drivers are working now. Some of them were lazily titled and would need to be cleaned up first. I probably wouldn’t submit the LCD controller one though — more on that in a bit.

The main thing dissuading me from submitting most of these is that when I first tried to submit those top two patches, they went completely ignored. I had to go out of my way to pester the SoC subsystem, not even marked as the actual maintainer, to get them accepted. If my initial experience dealing with this stuff had gone the same way as all the other subsystems, I probably would have put more effort into cleaning most of these up and submitting them upstream.

I’m not upset about any of this though. I think the reason for my trouble here is this is an old ARM SoC and there isn’t much interest in it. I feel like if I were doing the same thing for a more modern system that is still actively maintained, it would be a much more positive experience. Who knows, I might eventually muster up the motivation to clean up and submit some of these just to show the world how many peripherals are working in the PXA16x now.

pwm

Mainlined commits (series V1, V2, V3):

Working on this driver was a pretty fun experience. My goal here was to get the PWM backlight for the display working correctly. It went through a few rounds of code review. The existing code had a serious issue that didn’t affect me and I wasn’t even trying to fix, but ended up fixing anyway during the review process: it wasn’t resetting the duty cycle to 0 when you stopped it, so if you tried to restart it with a different duty cycle, it would still have the old setting for a brief moment. You can’t configure it without clocking it, and as soon as it’s clocked it will start outputting whatever you last had it configured for.

mmc

Mainlined commits (series V1, V2, V3, V4, V5):

I really enjoyed working on the SDHCI driver as well. It’s fully working in the mainline kernel and I’m proud of it. This was another case where at least one old patch already existed, but for some reason Marvell never pushed it across the finish line. Coming up with the IRQ workaround on my own (with the help of a suggestion from the SDHCI maintainer) was pretty enjoyable too.

The most recent commit is from last month and involved fixing a static analysis warning that was reported to me. I was unnecessarily checking if a pointer was null in a place where it’s already known to not be null. It is likely to be included in Linux 6.11 since it was accepted into mmc’s “next” branch. I only linked to my patch submission because it’s not in Linus’s tree yet as of this writing.

wifi

Mainlined commits (series V1, V2, V3, V4):

This was a fun driver to work on too. I fixed a serious problem with the libertas driver that prevented it from working with modern versions of wpa_supplicant. Now it works great! I also took the opportunity to improve WPS support, at the suggestion of the driver’s previous maintainer.

serial

Mainlined commits:

These two commits are actually very recent and were discussed in part 12. I fixed two different drivers for the PXA UART that were failing in different ways. Now they both work fine. In the case of the second “revert” commit, my V2 series also included two other patches developed by the original author of that commit to fix the original issue in a different way that wouldn’t break the pxa and omap serial drivers.

I haven’t submitted anything upstream for the rest of the subsystems I’m going to talk about (with one exception where my patch was ignored):

power

Commits only in my branch:

This is a driver that enables the reboot and shutdown commands to work properly by forwarding those requests onto the STM32 coprocessor. I added the new driver, and more recently I realized that I had accidentally coded it in a way that made it try to take exclusive control of the UART’s I/O range, but the UART driver also needs access to it during normal system operation. Without that last fix, this driver wouldn’t load alongside the 8250_pxa driver because the 8250_pxa driver had already taken over the I/O range. I switched to using ioremap() so I could ignore the conflict.

I would probably squash these two commits together into one if I attempted to upstream this driver. I didn’t bother submitting it upstream because it’s only useful for the Chumby and I don’t even have a board file for the Chumby in the mainline kernel. I think I would only submit this if I got to the point that there was actually a Chumby device tree board file upstreamed.

spi

Commits only in my branch:

These would probably both be easy to submit. I would need to add a commit updating the DT bindings though. The PXA168’s SSP peripheral is almost the same as the existing one supported by this driver, but it doesn’t have support for a clock divider, so the driver needs to be told about that. The DMA size thing ensures that we don’t waste a bunch of time setting up DMA if there’s a transfer of only a few bytes. This drastically improved performance with the touchscreen driver I ported/created. Speaking of which:

touchscreen

Commits only in my branch:

I haven’t submitted this upstream for the same reason as the power/reset driver. Without a Chumby .dts file being in the kernel, I feel like it wouldn’t make much sense. I guess I would get the benefit of the driver being updated by others whenever something about the API changed, but I’m not sure it’s worth it. If this upgraded Chumby kernel had more than a few users, maybe I would consider it.

drm

Commits not created by me (and not mainlined) that I depend on:

Commits only in my branch:

The armada driver hasn’t been updated in mainline for device tree support. Russell King actually made progress on this, but it seems like his changes slipped through the cracks and were never merged. I found these commits in his ARM linux tree on armlinux.org.uk. I feel like most of these commits would be worth submitting upstream (maybe not the “preserve existing framebuffer” and “automatically enable the peripheral clock” commits though), but I can’t because they’re useless without the DT support that Russell’s commits added.

I suppose I could submit them myself as part of a series, but I think it would be a lost cause. I actually have tried multiple times to submit that last GEM object fix to mainline, but it has been ignored every time. I might be doing it wrong, but it fixes a bug I ran into that prevented video playback from working. I think this driver is in a weird state where it’s marked as being maintained but it’s not really being maintained. For that reason, these changes will likely remain in my branch for the foreseeable future.

That’s a summary of everything that I accomplished as part of this kernel upgrade project. What do I actually have to show for it now? Well, I have a Chumby 8 and an Insignia Infocast NS-DP8CH that both start up with a Chumby logo, eventually go to a white screen, and then sit forever waiting for me to do something.

It doesn’t look like much, but lots of stuff is going on under the hood. It loads an X server, connects to Wi-Fi, syncs up with an NTP server, saves the date and time, and is capable of playing small simple videos, playing and recording audio, displaying graphics, and recognizing touchscreen presses. I could make a simple Qt app for something like a clock or displaying pictures or whatever, and it would work fine with my modern kernel. I haven’t decided what I want to do with it yet!

Let’s finish this post off by looking at how I got to this point, even before the Chumby 8 was a thing. I’ve been tinkering with embedded ARM Linux devices since around 2007. Back then, I was working on a product that used the Cirrus Logic EP9302 and ran Linux 2.4. Yes, even though Linux 2.6 had been out for years at that point, I was still using 2.4 because it was what the board vendor provided and it was what I had to work with. I was also very much a newbie so I didn’t have the knowledge required to upgrade to something newer.

In 2011, I started developing a different product that used the Texas Instruments AM3517. Unfortunately, TI didn’t do much toward mainlining drivers on these chips, so once again I was stuck with an old vendor kernel version: 2.6.37. I think the work I did on this product was what really started piquing my interest in playing with embedded Linux at home, which is why I originally bought my Chumby 8.

When I found out the Chumby was also running a custom old vendor kernel (2.6.28), I think this was my breaking point where I started wondering why everybody was using vendor kernels instead of getting everything working in the upstream kernel. I was genuinely curious. In 2013 after I was a little more seasoned with my AM3517 experience, I started tinkering with the Chumby to see if I could understand how much effort it would be to run a newer kernel (3.13-rc3 at the time).

Why didn’t my Chumby kernel upgrade attempt in 2013, back when there were more users and it was still a fairly new architecture, really get anywhere? I think there are a number of reasons. Most importantly, I just didn’t have the time to truly commit to it. If I had infinite time, I feel like there would be so many cool things I could accomplish. But time is our most precious resource and we have to spend it wisely.

Another reason I failed in 2013 was because it was right in the middle of the process of everything being converted to device tree. I didn’t understand anything about device tree at the time. All of my professional projects didn’t involve it. I was doomed to fail until I gained a little more experience, and I think I needed to wait for things to stabilize a little more in the kernel anyway.

By the time 2022 rolled around, I had way more experience in all facets of embedded Linux development. I had learned that compiling all of your packages by hand is insane and you should really be using a build system like buildroot or Yocto to do it for you. I had also worked extensively with device tree at this point on multiple products and architectures, including the Xilinx/AMD Zynq-7000, ST STM32MP1, and also tinkering with QEMU. I still didn’t know much about submitting kernel patches, but I had successfully sent a simple patch for adding a device ID to the ftdi_sio driver in 2016 and an AppleTalk fix in 2021.

One more thing that I think was vital to this project’s success was gaining a better understanding of Git. In 2013 I didn’t really know what I was doing with Git. I knew about adding and committing and pushing, but I was mostly using Subversion at work. I feel like in order to do Linux kernel development it really helps to have a good foundation for how to use Git. In particular, the concepts that really changed my life were learning how to use “git add -p” to stage smaller chunks of my changes at a time and “git rebase -i” to interactively rearrange and squash commits after the fact. Before I got comfortable with those two commands, I never really felt like I understood how to tell Git exactly what I wanted to do.

So in summary: in order to succeed with this project, I needed more experience in the kernel, a better understanding of device tree, a better build environment, and a lot more experience with Git. With all of these skills gained, I was finally ready to start actually pushing forward with getting these fixes upstreamed.

As you have seen above, I still have a bunch of commits that live only in my fork. For a cleaner view through GitHub, here are the commits from my fork’s Linux 6.10 branch. Essentially I have created my own vendor kernel at this point! My whole goal was to mainline things, and where I’ve left off is I have my own vendor fork. This kind of seems like the opposite of what I was trying to accomplish.

So how do I feel about vendor forked kernels versus mainline kernels after doing this project? To be honest, I think forks are inevitable. At the very least, the chip manufacturers need a place to stage their changes prior to submitting them upstream, so they will always exist in some form. Additionally, it takes a lot of extra work to get your changes accepted in the mainline kernel. You have to go back and forth with maintainers and reviewers to ensure your code meets their requirements, which is actually an amazing experience and definitely increases the code quality of the kernel, but it also means the success of your patches is tied to the schedule of the busy kernel developers.

I can see why some vendors have historically skirted the process and just kept their own forks. My outside opinion is that the mainline situation now is definitely better than it was in 2013, but I’m not sure I’m qualified to comment. It also probably depends on which vendor’s chips you’re using. Ideally it makes sense that the vendors should be striving toward making their forks totally unnecessary as everything gets merged upstream, but as we can see it doesn’t always happen in practice. I wish it did though!

With some of the hacky stuff I did like my changes to the audio driver, I felt it would have been a nightmare to go back and try to do it correctly so that it would be accepted upstream. Especially since the driver is also used on other PXAxxx SoCs. How would I know for sure that my changes were correct for all SoCs and not just mine? How would I justify reverting a previous decision about how the PXA audio driver works? Is “I couldn’t figure out why the PXA166 was outputting a screeching sound and going back to the old method fixed it” a good enough reason to give to the maintainer? At some point the effort probably wouldn’t be worth it considering how unpopular these SoCs are — both from my standpoint and the maintainer’s standpoint. I don’t want to waste anyone’s time.

If this were a more popular architecture, it would make sense that the vendor would still be actively supporting it and figuring out what was really causing such a problem so they could confidently state a root cause. They would be capable of making sure it worked correctly on all SoC variants that use the driver. In my case, I’m just a guy who has one board that uses this chip, so I don’t have the resources to figure that all out by myself. It’s also just one small piece of a much larger project so would I really want to spend weeks on it?

I guess the overall point I’m trying to make here is that for newer, actively-used architectures, I can see how it makes perfect sense for the SoC vendors to upstream everything that they can. Now that this particular Marvell chip is well over a decade old, doesn’t have many users, and the employees who worked on it have moved onto different companies, it is definitely more difficult to get changes tested and accepted in certain places. Like I said, I don’t think the effort required to fully mainline everything would actually be worth it for an old, practically dead, SoC.

That leads me to the last big thing I want to discuss. I do want to point out that this old Chumby won’t necessarily be supported by Linux forever. Older ARM variants end up being a pain to support not only for the ARM kernel developers, but also for the GCC developers. Here’s an interesting recent post to the linux-arm-kernel mailing list discussing when support for various old architectures and features can be dropped. It doesn’t look like the ARMv5TE used in the PXA168 is at risk of being dropped in the immediate future, but the iWMMXt instruction set is slated to go away soon because it’s not known to have any users, not to mention it was marked as deprecated in GCC 14 and will likely be removed in GCC 15. I don’t think this will really change anything for my purposes. As far I know I’m not using it. If I had decided to put more effort into improving H.264 decoding performance, though, it’s likely that this deprecation plan would have interfered with it. I’m glad I didn’t attempt anything!

In conclusion, I still think this project was totally a success even if every commit wasn’t mainlined. Some of it was out of my control. Even so, I still fixed many issues in the mainline kernel, developed a few new drivers in my fork, and obtained a ton of valuable experience submitting patches. I definitely have a working modern development foundation for my Chumby 8 even if it doesn’t really do anything yet. I guess now I need to make some real userspace software for it, and sit back and enjoy it!

Address: https://www.downtowndougbrown.com/2024/08/upgrading-my-chumby-8-kernel-part-13-the-end/
«
Trackback

4 comments

  1. Ben Stern @ 2024-08-06 11:40

    Thanks for all of your hard work! I look forward to running this kernel on my various Chumby 8 units!

    Now, as you say above, it’s time to figure out what userland stuff I want to run on them.

  2. Doug Brown @ 2024-08-06 15:24

    I’m excited to hear how it goes for you Ben! I plan on at least trying to keep my fork up to date as new kernels come out, at least for the foreseeable future.

    One thing I forgot to add in my final post is links to the repositories containing my work:

    – buildroot – use chumby8_defconfig. This is really the main repo needed for people who want to play; it pulls everything else down automatically when built.
    – linux
    – u-boot
    – chumby-utils – contains the card reader daemon
    – xf86-video-armada – my fork of the X11 plugin, with a few bug fixes
    – libdrm-armada – just a mirror of the original
    – Chumby8TSCal – touchscreen calibration program

  3. ma_jk @ 2024-08-09 02:45

    That was an epic journey!
    Now… could you attempt to update also the Chumby One? You would make a lot of people happy.

  4. Doug Brown @ 2024-08-09 19:13

    Thanks ma_jk!

    I actually do have a couple of Chumby Ones that have been sitting around gathering dust. Maybe someday, but realistically I’m not sure I see myself working on it anytime soon. It looks like there was already some progress on getting it working with kernel 4.x a few years ago, so with any luck it might be less work: https://forum.chumby.com/viewtopic.php?id=10100

Add your comment now