Downtown Doug Brown

A couple of weeks ago I found a really good deal on a Socket AM4 motherboard that supports the newest AMD Ryzen CPUs. The motherboard is an ASRock A520M/ac. It’s a very basic motherboard which doesn’t appear to be sold by any of the usual retailers anymore, but I couldn’t pass up on the deal, especially with the potential it had for being a fun learning project.

The reason I got such a good deal on it was because it was sold in non-working condition, but the seller and I both had a pretty good hunch about what was wrong. The seller said that they had bought it as an open box unit, but couldn’t get it to POST. However, they had only tried CPUs in it that were not compatible with the original BIOS version. I decided to have some fun and see if that was indeed the only problem. I didn’t have an older CPU available to easily test that theory. I did have a new Ryzen 7 5700G, which is only supported by BIOS revision P1.60 or newer.

Typically, there are several simple options for using a newer CPU with a motherboard that needs a BIOS update in order to support it:

1. Borrow an older CPU just long enough to install an updated BIOS. AMD has a program for handling this if you don’t have an easier way to borrow one. I don’t know if this is a valid option if I’m not the original buyer of the motherboard. AMD’s documentation requirements in order to participate seem pretty stringent based on the linked instructions.
2. Use the “USB BIOS Flashback” feature to update the motherboard’s BIOS even without a CPU installed. This particular motherboard doesn’t support that option.
3. Send it back to the retailer or manufacturer to update it for you. I have no idea which retailers/manufacturers might do this. There’s no way that Amazon, for example, would provide this service.

It’s possible that ASRock would have tried to help me out if I had asked, but I decided to turn this into a fun personal challenge instead: upgrade the BIOS on my own without using an older CPU.

Here’s a quick post — it should be a nice short one by my standards. This weekend I decided to upgrade a couple of my Ubuntu servers from 18.04 to 20.04. I ran into a bit of a problem with a really tiny cheap VPS that I keep mainly for playing around. It only has 256 MB of RAM and 5 GB of storage. It was an interesting challenge finding enough free disk space to complete the upgrade process to begin with, but that ended up being the easy part.

I’ve been involved a little bit with the process of porting RPiPlay to run on desktop Linux. RPiPlay is a program originally designed for the Raspberry Pi that acts as an AirPlay Mirroring server and supports mirroring your iOS device’s screen to your Raspberry Pi’s video out. Originally it only supported the Raspberry Pi, but antimof reworked the code to also work on desktop Linux with GStreamer, and I helped get it across the finish line and merged back into the main project.

A while ago, I noticed that when I ran RPiPlay in a VMware virtual machine during development, the video was messed up. It looked like some kind of horizontal synchronization issue. The image looked like it was stretching out further and further to the right on each successive line.

It worked fine on my laptop running Linux directly, which is probably the way most people use RPiPlay, so I didn’t think much more about it at the time. It bothered me though. It seemed to be a problem at a level deeper than RPiPlay, and I really wanted to understand why it was happening. So of course, I recently dug myself deep into a rabbit hole to try to figure it out.

I wanted to share a story of a segmentation fault I helped track down this weekend. I thought the final root cause of the segfault was interesting because of how unrelated it was to the code I was trying to debug.

I’ve been maintaining a Linux fork of obs-ios-camera-source, which is an OBS plugin that allows you to use an iPhone or iPad’s camera and microphone as a video and audio source in OBS. It works in conjunction with the “Camera for OBS Studio” app in the App Store. This kind of thing is useful for online streamers who want to use their phone’s camera instead of buying a separate camera. For those of you who don’t know, OBS is short for Open Broadcaster Software. A lot of streamers use it to handle broadcasting their stream. It allows you to capture audio and video, mix it all together, do all kinds of cool things with it, and then record the final result and/or stream it to sites such as YouTube and Twitch.

Getting this plugin working on Linux wasn’t really complicated, because it was already well-written without much platform-specific code. After all, the existing codebase was already operational on both macOS and Windows. It mostly just required tweaking a few compile/link options to make the code run happily on Linux.

Anyway, I’m pretty sure a good number of people have been using my Linux port of this plugin without issues. I know it works fine for me when I test with it in Ubuntu 18.04 or 20.04. I’ve helped people on other distros get it working too. I don’t really do any streaming myself — maybe someday though!

On Friday, GitHub user rrondeau reported an issue: after a half a year of the obs-ios-camera-source plugin working without a problem, it suddenly started causing OBS to segfault on his computer (currently running Fedora 33). He provided a stack trace that showed that the segfault was happening because of something initiated by the plugin. Afterward, he used GDB to get a better stack trace that provided more info about the functions being called and the parameters being passed:

#0  0x00007fffee7abc64 in socket_send () at /usr/lib64/samba/libsamba-sockets-samba4.so
#1  0x00007fff88b7813c in send_packet (sfd=50, message=8, tag=1, payload=0x1b22e60, payload_size=488) at /home/rrondeau/git/perso/obs-ios-camera-source/deps/libusbmuxd/src/libusbmuxd.c:400
#2  0x00007fff88b782a6 in send_plist_packet (sfd=50, tag=1, message=0x1ae53e0) at /home/rrondeau/git/perso/obs-ios-camera-source/deps/libusbmuxd/src/libusbmuxd.c:431
#3  0x00007fff88b7851b in send_list_devices_packet (sfd=50, tag=1) at /home/rrondeau/git/perso/obs-ios-camera-source/deps/libusbmuxd/src/libusbmuxd.c:499
#4  0x00007fff88b79367 in usbmuxd_get_device_list (device_list=0x7fffffffc740) at /home/rrondeau/git/perso/obs-ios-camera-source/deps/libusbmuxd/src/libusbmuxd.c:938
#5  0x00007fff88b725e1 in portal::Portal::addConnectedDevices() (this=0x1909378) at /home/rrondeau/git/perso/obs-ios-camera-source/deps/portal/src/Portal.cpp:109
#6  0x00007fff88b72684 in portal::Portal::reloadDeviceList() (this=0x1909378) at /home/rrondeau/git/perso/obs-ios-camera-source/deps/portal/src/Portal.cpp:126
#7  0x00007fff88b722db in portal::Portal::Portal(portal::PortalDelegate*) (this=0x1909378, delegate=0x1909240) at /home/rrondeau/git/perso/obs-ios-camera-source/deps/portal/src/Portal.cpp:57
#8  0x00007fff88b67053 in IOSCameraInput::IOSCameraInput(obs_source*, obs_data*) (this=0x1909240, source_=0x1aee000, settings=0x19210a0)
    at /home/rrondeau/git/perso/obs-ios-camera-source/src/obs-ios-camera-source.cpp:74
#9  0x00007fff88b66358 in CreateIOSCameraInput(obs_data_t*, obs_source_t*) (settings=0x19210a0, source=0x1aee000) at /home/rrondeau/git/perso/obs-ios-camera-source/src/obs-ios-camera-source.cpp:371
#10 0x00007ffff6259c2a in obs_source_create_internal () at /lib64/libobs.so.0
#11 0x00007ffff626bb81 in obs_load_source_type () at /lib64/libobs.so.0
#12 0x00007ffff626e3c2 in obs_load_sources () at /lib64/libobs.so.0
#13 0x000000000049e750 in OBSBasic::Load(char const*) (this=0xa370b0, file=0x7fffffffd040 "/home/rrondeau/.config/obs-studio/basic/scenes/Untitled.json")
    at /home/rrondeau/git/perso/obs-studio/UI/window-basic-main.cpp:973
#14 0x00000000004a2976 in OBSBasic::OBSInit() (this=0xa370b0) at /home/rrondeau/git/perso/obs-studio/UI/window-basic-main.cpp:1783
#15 0x000000000047feff in OBSApp::OBSInit() (this=0x7fffffffd690) at /home/rrondeau/git/perso/obs-studio/UI/obs-app.cpp:1415
#16 0x0000000000482503 in run_program(std::fstream&, int, char**) (logFile=..., argc=1, argv=0x7fffffffdd68) at /home/rrondeau/git/perso/obs-studio/UI/obs-app.cpp:2052
#17 0x0000000000484203 in main(int, char**) (argc=1, argv=0x7fffffffdd68) at /home/rrondeau/git/perso/obs-studio/UI/obs-app.cpp:2697

The actual segfault was happening inside of a function called “socket_send” in libsamba-sockets-samba4.so, which was being called by a function in libusbmuxd, which is bundled as part of the obs-ios-camera-source plugin source code and is used for communicating with iOS devices over USB. When I first saw this in the stack trace, my mind thought “Huh…that’s weird. Why does libusbmuxd use Samba’s library for its socket code instead of providing its own?” (Samba is an implementation of the Windows file sharing protocol used by pretty much every Linux distribution)

I tested and couldn’t reproduce the issue in Ubuntu. I know basically nothing about Fedora, but I faked my way through grabbing a Fedora 33 virtual machine, installing OBS, and compiling the plugin. I ran into the exact same issue that he was seeing.

Before I had a chance to look deeper and understand what was going on, rrondeau beat me to the correct conclusion: code in Samba’s library was mistakenly being called. libusbmuxd has a function called socket_send, but clearly libsamba-sockets-samba4’s function that is also named socket_send was accidentally being called instead.

Honestly, that’s all we really needed to know. Renaming libusbmuxd’s socket_send function to something else, and updating all references to it to use the new name, fixed the issue. I still wanted to understand why this suddenly became an issue when it had been working fine prior to that. Why were we calling into Samba libraries? Why does an iOS USB multiplexing library even consider talking to a library associated with Windows file sharing?

Not knowing the answer to that question bothered me. I decided to dig deeper and understand exactly what was going on. I started by using ldd, which lists all dynamic libraries used by a program or library:

[fedora@fedora33 build]$ ldd obs-ios-camera-source.so 
	linux-vdso.so.1 (0x00007fffa599a000)
	libobs.so.0 => /lib64/libobs.so.0 (0x00007f0a3f688000)
	libavcodec.so.58 => /lib64/libavcodec.so.58 (0x00007f0a3e2db000)
	libavutil.so.56 => /lib64/libavutil.so.56 (0x00007f0a3e036000)
...
	libsamba-sockets-samba4.so => /usr/lib64/samba/libsamba-sockets-samba4.so (0x00007fd6af4b7000)
...

I truncated the output because it spit out a very long list of libraries. As we can see from ldd’s output, obs-ios-camera-source.so depends on libsamba-sockets-samba4.so. ldd lists all recursive dependencies as well, and I couldn’t find any references to “samba” in the plugin source code, so this was likely an indirect dependency instead. I confirmed this by using readelf to show only the direct dependencies:

[fedora@fedora33 build]$ readelf -d obs-ios-camera-source.so | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libobs.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libavcodec.so.58]
 0x0000000000000001 (NEEDED)             Shared library: [libavutil.so.56]
 0x0000000000000001 (NEEDED)             Shared library: [libobs-frontend-api.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

At this point I used ldd and readelf to walk through the tree of dependencies and figure out what was actually linking against the Samba libraries. I later learned that I could have installed lddtree (part of the pax-utils package) to do this automatically. Either way, this led me to discover that the Samba libraries were being included through libsmbclient, which was a dependency of libavformat (part of FFmpeg). libavformat is a dependency of libobs.

Repeating this experiment on Ubuntu showed that libavformat on Ubuntu does not depend on libsmbclient. This explains why I couldn’t reproduce the issue on Ubuntu. So why does Fedora’s (well, RPM Fusion‘s) version of libavformat depend on libsmbclient?

It turns out that it’s a compile-time option for FFmpeg. libavformat contains code for talking with Windows servers using libsmbclient, but it’s an optional thing that you can choose to enable at compile time. Clearly Ubuntu chooses not to enable it, but RPM Fusion does. Actually, I found the exact post on RPM Fusion’s commits mailing list where the patch was added for enabling SMB support in FFmpeg. This patch is what led to the whole issue happening. If Ubuntu’s version of FFmpeg was being built with SMB support, we would have seen this a long time ago. This commit to RPM Fusion was made on December 31, 2020, which explains why rrondeau had only recently begun seeing the problem.

The root cause here is that the obs-ios-camera-source plugin was linking against two libraries that both provided a function named socket_send: libsamba-sockets-samba4 (indirectly through libobs) and libusbmuxd. libusbmuxd was being linked statically, but that doesn’t prevent functions in it from being resolved through dynamic linking rules anyway. So even though libusbmuxd was a static library with its own internal implementation of socket_send, it was using libsamba-sockets-samba4’s implementation instead.

rrondeau and I settled on changing what we had control over: the libusbmuxd source code embedded inside of the plugin’s source code. We went with simply adding a “usbmuxd_” prefix before all of the socket_ functions. There may be a more complex way of forcing it to use its own internal version of socket_send through linker options, but I feel that this is probably the simplest solution. It’s easy to implement, and it gets the job done.

This segfault turned out to be a pretty simple issue to solve and diagnose. Is it really worthy of a blog post? Maybe, maybe not. I could definitely foresee someone else running into this issue with another combination of libraries. socket_create, socket_close, socket_send, etc. are such generic names that it may happen again. This is a great opportunity to remind everyone: don’t use generic function names like this in your shared libraries, at least not in your exported symbols! You could easily run into a situation similar to this one. In my opinion, prefixes are definitely a good idea for your library’s exported symbols. In this case, both libusbmuxd and Samba were breaking that guideline.

This can be tricky because dynamic libraries on Linux export all symbols by default unless you specify otherwise. This is backwards from how Windows works with DLLs. Windows DLLs require you to specify which functions are being exported. I actually like that approach better! Here’s an interesting reference on how to customize the visibility of your Linux dynamic library’s symbols.

libusbmuxd already fixed this on their end quite a while ago — they now only export functions intended to be public, which have a usbmuxd_ or libusbmuxd_ prefix. I think the version included with the plugin’s source code is quite a bit older. For fun, I tried applying the visibility fixes from the linked patch to the plugin’s embedded libusbmuxd source code. The patches don’t apply cleanly because the embedded libusbmuxd code is actually built using CMake, so I have to add the compiler flags to CMakeLists.txt. After doing that, it does indeed cause libusbmuxd’s internal socket_send function to be called instead, and thus fixes the segfault.

What do you think? Would it make sense to try to convince the Samba project to rename their exported socket functions, or would I be barking up the wrong tree? I suspect that Samba’s socket library is actually intentionally exporting these functions so that other Samba libraries can call the socket functions. Would renaming Samba’s exported socket functions to give them less generic names cause a ton of incompatibilities given how long those function names have existed? Is it too late at this point? Am I wrong to think that Samba’s exported socket functions should have a “samba_” prefix or something like that?

I like my EarPods. Yeah, not the new fancy wireless ones. Just the standard wired earbuds that have come with iPhones for a long time. At one point I realized I prefer to use my EarPods during meetings. I was actually using my iPad instead of my computer to join meetings simply because I could use the EarPods.

I was looking at one of my classic Macs a few weeks ago, and noticed that my Ubuntu 18.04 netatalk server wasn’t showing up in the Chooser anymore. If you’re not familiar with netatalk, it’s an implementation of Apple Filing Protocol (AFP) that runs on Unix-like operating systems such as Linux and NetBSD. It allows other operating systems to act as Mac file servers. Version 2.x, which I use, supports the ancient AppleTalk protocol. This allows it to work with really old classic Macs that don’t even have a TCP/IP stack installed. Support for AppleTalk was removed in version 3.x, so that’s why I’m still using 2.x.

I checked out the server, and noticed that atalkd wasn’t running.

doug@miniserver:~$ ps ax | grep atalkd
3351 pts/0 R+ 0:00 grep --color=auto atalkd

Hmmm….why wouldn’t atalkd be running? I went ahead and tried to restart netatalk:

A while ago, I put 16 GB of RAM into one of my computers. The computer is using a Foxconn P55MX motherboard with a Core i5 750. It’s old and could probably be replaced, but it still works for what I need.

Here’s the interesting part. This motherboard doesn’t officially support 16 GB of RAM. The specs on the page I linked indicate that it supports a maximum of 8 GB. It only has 2 slots, so I had a suspicion that 8 GB sticks just weren’t as common back when this motherboard first came out. I decided to try anyway. In a lot of cases, motherboards do support more RAM than the manufacturer officially claims to support.

I made sure the BIOS was completely updated (version 946F1P06) and put in my two 8 gig sticks. Then, I booted it up into my Ubuntu 16.04 install and everything worked perfectly. I decided that my theory about the motherboard actually supporting more RAM than the documentation claimed was correct and forgot about it. I enjoyed having all the extra RAM to work with and was happy that my gamble paid off.

Then, a few months later, I tried to boot into Windows 10. I mostly use this computer in Linux. I only occasionally need to boot into Windows to check something out. That’s when the fun really started.

After I upgraded Ubuntu MATE from 16.04 to 18.04, I noticed a really annoying behavior. Clicking in the empty area of a scrollbar (between the handle showing where your current position is and the up or down arrow) used to scroll up or down a page at a time. After the 18.04 upgrade, this behavior changed, and now it moves the handle directly to the position where you clicked. Some people like this, some people don’t. I don’t. Unfortunately, they didn’t decide to make this a configurable option in the graphical settings for some strange reason.

It turns out that this is actually a GTK 3 behavior — it’s caused because MATE recently changed to use GTK 3. Luckily, it’s an easy fix:

Go into the directory of the theme you are currently using. So for my current configuration, I went into this directory:

/usr/share/themes/Ambiant-MATE/gtk-3.0

Edit the “settings.ini” file. Add the following line to the bottom of it:

gtk-primary-button-warps-slider = false

Log out and back in, and you’re done.

After upgrading my machines from Ubuntu MATE 16.04 to 18.04, I noticed that something was subtly wrong with my top menu/panel: the network notification icon was missing. Something about the 16.04 to 18.04 upgrade process broke it. It happened on both of the machines I upgraded, so I don’t think it was a random thing that only I was experiencing.

Note: my fix for this will reset your panel layouts, so take a screenshot and/or save any info about the layout of your panels before following the steps below so you will have a reference for restoring it back the way you want it afterward.

To start out, I will share some background info: if you are used to the default menu layout of the top panel with the Applications, Places, and System menus, they decided to change this in 18.04. That layout is called “Traditional”, and Ubuntu MATE 18.04 now defaults to a new layout called “Familiar”. I personally don’t like the Familiar layout because it doesn’t have the Places menu, but I guess enough people liked it that they wanted to make it the default.

Anyway, the fix for this is to reset your panel to one of the default configurations, which will add everything back properly. But like I said in bold above, it will also erase any customizations you have done. Here’s the fix:

1. Open up MATE Tweak. In the Traditional layout, it’s in the System -> Preferences -> Look and Feel menu.
2. In MATE Tweak, click on the Panel category.
3. In the top dropdown menu, choose the layout you want. To match Ubuntu 16.04’s default, choose Traditional. To match Ubuntu 18.04’s default, choose Familiar.
4. Confirm that you want to change the layout by clicking OK.
5. All done! Your network notification icon should be restored, and now you can reapply any of your customizations to the panel.

Another way to fix it is to right-click on the panel and choose Reset Panel, but I think the method I described above is better because it lets you pick which layout you want to start from.

I recently upgraded a couple of my machines from Ubuntu MATE 16.04 to 18.04. On both of them, I noticed that after the upgrade, the login window did not look correct. There was no background image. It didn’t look like a fresh 18.04 install. Plus, any changes I did in the Administration -> Login Window program didn’t take effect. I did some digging, and figured out what was going on.

Sometime between 16.04 and 18.04 (it appears that this happened starting in 17.10), Ubuntu MATE decided to stop using lightdm-gtk-greeter, and instead switched to using slick-greeter. It seems to be that on update installs, lightdm-gtk-greeter is not being removed, so it’s defaulting to using it rather than slick-greeter.

I was able to solve this very easily by running the following command:

sudo apt purge lightdm-gtk-greeter

After that, I logged out, and Ubuntu MATE’s correct login screen appeared. It’s as simple as that!