Dec
08

USB mass storage with embedded devices: tips and quirks

Doug Brown Uncategorized 11 comments

I work on embedded devices that have the capability of installing a firmware update by plugging in a USB flash drive containing an update file. These devices can also save reports onto an attached flash drive. Historically, these devices have worked with the various drives I’ve been able to test in house, but there have been occasional reports of incompatible drives in the field. I just used the sample code provided with the microcontroller manufacturer’s USB library, so I had no idea what I could do to improve compatibility.

Sometimes the problem is outside of my control. If the drive is larger than 32 GB, Windows 10 formats it as exFAT, and I don’t currently have exFAT support enabled (it’s too expensive to license from Microsoft). But most of the time, the problem isn’t the filesystem. The problem is that each drive behaves a little differently.

I decided to dedicate some time to improving USB drive compatibility in the embedded devices I work on. I researched USB mass storage devices, the USB specification, and the SCSI protocol that flash drives speak over USB. I bought an old Ellisys USB Tracker 110b, which is capable of recording raw USB 1.1 traffic (this is suitable for my needs, because the embedded devices I work on are only capable of full speed). Then, I bought a ton of used flash drives on eBay. My goal was to try as many drives as possible and discover various quirks. I also got a USB 1.1 hub in order to limit the drives to full speed on modern computers, and recorded what happened when I plugged the drives into Windows XP, Windows 10, Linux, Mac OS X, and Mac OS 9. 

I was successful. I found lots of little differences in how USB drives work. The purpose of this post is to share my findings to help others in the future who might have trouble with USB flash drive support in their embedded products.

Specifications

Here is a list of links to relevant specifications that will be useful as a reference:

Brief overview of how USB mass storage works

Before I get into specific details, I want to start with a quick overview of how all of the protocols explained in the specifications above combine together to enable computers to communicate with flash drives over USB.

When a flash drive is plugged in, the computer looks at its device, configuration, interface, and endpoint descriptors to determine what type of device it is. Flash drives use the mass storage class (0x08), SCSI transparent command set subclass (0x06), and the bulk-only transport protocol (0x50). The specification indicates that this should be specified in the interface descriptor, so the device descriptor should indicate the class is defined at the interface level.

What does this all mean? It just means that there will be two bulk endpoints: one for sending data from the host computer to the flash drive (OUT) and one for receiving data from the flash drive to the computer (IN). Data sent and received on these endpoints will adhere to the bulk-only transport protocol specification linked above. In addition, there are a few commands (read max LUN and bulk-only reset) that are sent over the control endpoint.

The host starts out by sending a 31-byte command block wrapper (CBW) to the drive, optionally sending or receiving data depending on what command it is, and then reading a 13-byte command status wrapper (CSW) containing the result of the command. The CBW and CSW are simply wrappers around Small Computer System Interface (SCSI) commands. Descriptions of the SCSI commands are available in the last two specifications I linked above.

That’s all there is to it…except I haven’t said anything about which SCSI commands you’re supposed to use, or when. SCSI is a huge standard. Reading the entire standard document would take a ridiculous amount of time, and it wouldn’t really help you much anyway. Unfortunately, the standards don’t provide a section entitled “recommended sequence of commands for talking to flash drives over USB”.

This is where I originally hit a roadblock when I was implementing USB support, and it’s also why I simply stuck with the sample source code provided with the USB library I used. Unfortunately the sample source code was not good enough. What it comes down to in practice is you should try to do something similar to what Windows does, because pretty much every flash drive is compatible with Windows.

Initialization sequence

The first important thing to do when a USB flash drive is detected is to figure out information about it. Is it actually a flash drive? How big is it? How many logical units (LUNs) does it have? I found that if I didn’t follow a sequence with some preliminary commands that operating systems do, a SanDisk drive with a bunch of files on it would crash when I first attempted to write to it. Interestingly, the same drive didn’t have the same problem when it was empty. You may be thinking I’m an idiot and a problem like this obviously has to be in the filesystem library and not the USB library, but I swear that the problem was the USB communication to the drive itself, because imaging the drive to my computer with “dd” and using the same filesystem library on the raw drive image worked fine. The Ellisys USB Tracker confirmed the drive responded with a stall condition after the write, and after clearing the stall, it was hung up, even after a mass storage bulk-only reset command, which is supposed to prepare the drive to receive a new command.

Based on what I observed Windows, Mac OS X, and Linux doing, I changed my initialization sequence, and that problem completely went away. The sequence I do now is not an exact clone of any other OS, and it’s probably doing extra overkill commands, but hey, it works:

  1. Request the maximum LUN. If this request stalls, assume the maximum LUN is 0. Start working with LUN 0 in either case.
  2. Keep trying the sequence of “TEST UNIT READY” followed by “INQUIRY” until they both return success back-to-back. At this point you can look at the returned inquiry data to get more information about the name of the drive, if you care. For the inquiry, request 36 bytes. That’s what pretty much every OS does, so it’s best not to deviate from that.
  3. If the “INQUIRY” response data indicates that the peripheral device type is not 0 (meaning “direct-access device”), and there is more than one LUN, repeat step #2 with additional LUNs until you find one that is a direct-access device. Some promotional flash drives use LUN 0 as an emulated CD-ROM and the flash drive is on LUN 1, so you’d want to use LUN 1 instead of LUN 0 in that case. After this process is complete, use the LUN you discovered for all of the rest of your commands going forward. If you don’t find anything matching, just stick with LUN 0 in case the inquiry data is wrong.
  4. Attempt a “PREVENT ALLOW MEDIUM REMOVAL” command. A lot of operating systems do this, and most drives don’t support that command. It’s no big deal if the command fails. Just continue on. Interestingly I didn’t observe Windows XP sending this command on the drive I tested, but Windows 10, Mac OS X, and Linux did. I don’t know whether to include this command or not. It works for me.
  5. Keep attempting a “READ CAPACITY (10)” command until it succeeds. This will tell you the size of the drive in blocks (minus 1, because it returns the address of the last block), as well as the block size in bytes.
  6. Try a “MODE SENSE (6)” command, requesting 192 bytes of data on mode page 0x3F. The 192 matches what other operating systems request, so it’s best to match that. In the response data, if bit 7 of byte 2 is set, the drive is read-only. If the mode sense command fails, just move on. I haven’t found a drive that fails this command though.
  7. Just to be safe, do “TEST UNIT READY” again, repeating until it returns success. Now you are ready to send all the “READ (10)” and “WRITE (10)” commands you want to send.

In all of the places where I said to keep attempting something, I have a timeout of 5 seconds. If I don’t succeed and 5 seconds have elapsed in that step, I bail with an error. No drive I’ve tested so far has caused the 5 second timer to elapse, but if you’re extra worried you could try increasing the timeout.

Things to watch out for

As I tested various drives, I noted strange behaviors in certain cases. Here is a list of things you should probably watch out for.

Mass storage reset

The sample code that came with the USB library I use performed a “Bulk-Only Mass Storage Reset” command as the first step. None of the operating systems I tested did this, so I removed it. I think you should only use this command as a last resort if you have lost communication with the drive and it has stopped responding to your CBWs. (Make sure the drive isn’t simply waiting for you to read back a CSW after a stall or something too…)

Drives that are both a CD and flash drive

As I mentioned above, some promotional drives are both a CD-ROM and a flash drive. Check for that type of drive with the “Get Max LUN” command, and use the INQUIRY data on each LUN to find the one that’s actually the flash drive.

MODE SENSE (6) or MODE SENSE (10)?

While I was checking out various operating systems, I noted that sometimes Windows and Mac OS X use “MODE SENSE (6)” and sometimes they use “MODE SENSE (10)”. Linux seems to always do “MODE SENSE (6)”. I couldn’t figure out how Windows and OS X were making that determination.

I originally tried just always using “MODE SENSE (10)”, since I also always use “READ (10)” and “WRITE (10)”, so I figured why not the same with mode sense? However, that was a mistake. Some drives don’t support that command, and others return incorrect results in it. One drive I tested was particularly frustrating. Its “MODE SENSE (10)” response indicated that it was locked, even though it wasn’t. Its “MODE SENSE (6)” response correctly said it was not locked.

The moral of this story? Just stick with “MODE SENSE (6)”. Every drive I’ve tested supports it and returns correct-ish data. One drive I tested returns an incorrect data length as the first byte of the SCSI response data (the mode parameter header), so if the drive only returns 4 bytes but claims there are 70 in the response, you might want to limit your parser to only check the first 4.

As I said earlier, you should request 192 bytes of data in this command to match what other operating systems do. I’d recommend requesting mode page 0x3F, which means “all pages”. I’ve read online that some misbehaving drives may get confused if you send a mode sense request for any other page or data length.

First TEST UNIT READY command fails

On a lot of drives, the first “TEST UNIT READY” command returns failure. The sense data (obtained with “REQUEST SENSE”; see below) indicates it’s a temporary condition and to try again. On most of these drives, the next attempt succeeds. On one drive I tested, it failed the first 14 attempts.

This is why my initialization sequence says to keep trying “TEST UNIT READY” until it succeeds. I added “INQUIRY” in there as well because it seemed other OSes would intermix “INQUIRY” with it too. If after 5 seconds (or whatever time limit you’re comfortable with) it still hasn’t responded with success, then maybe something really is wrong.

Repeat failed commands

Maybe I should have just generalized the above section, but I’ll repeat it here. If a command fails that you really care about, just try again. I have it set up so if a “READ (10)” or “WRITE (10)” fails, I try again a few times before immediately bailing with an error.

In general, if an error occurs because the CSW indicates failure (and this rule of thumb also applies during the initialization sequence I described above), you should follow up with a “REQUEST SENSE” command to read information back about the failure before sending any other commands. Why? Because all the other operating systems do it too, so it’s a good idea. You are guaranteed that the drive has been tested under that behavior.

Theoretically the drive will tell you in the returned sense data whether the command failed due to a temporary condition or if the command is not supported. In practice, I do the “REQUEST SENSE” command and read the response from the device, but I ignore the content of the response. I simply repeat commands that I know are important, and I live with failure and move on if they’re not important (e.g. “PREVENT ALLOW MEDIUM REMOVAL”).

Write delays

This is probably obvious, but I’m pointing it out anyway. Sometimes “WRITE (10)” operations take a while to complete. When this happens, the drive will respond with NAKs until it’s finished. The NAKs could occur at any point — maybe while trying to read back the CSW, or while sending the next CBW, or in the middle of the data transfer process. If you’re designing a communication protocol that receives data and writes it to disk, make sure it has the ability to pause if the disk is too slow to keep up.

Handling short responses

In some cases, you will request data from the flash drive, but it will respond with less data than you requested. A perfect example of this situation is the “MODE SENSE (6)” command when requesting 192 bytes of data. I haven’t found a flash drive yet that has 192 bytes of mode pages to respond with.

According to the bulk-only transport specification, if the host requests more data than the device can provide in this situation, it’s allowed to pad the response with extra data to match the requested length. If it doesn’t do this, it must stall the BULK IN endpoint after transmitting as much data as it can. I was able to observe different drives that implemented each of these behaviors.

It turns out that there are some flash drives that ignore the above requirement and do it a different way. They don’t pad the response with extra data, and they don’t stall the endpoint. They simply send as much data as they can, without stalling the endpoint afterward. Although these devices aren’t following the bulk-only transport specification correctly, it’s not too hard to handle this situation. If you receive a USB data packet shorter than the endpoint’s maximum packet size (a “short packet”) when reading a response, but you haven’t received all the data you expected, you know the response has been terminated early by the drive, and the next read you attempt will give you the CSW. The USB library I use didn’t handle this case properly and would hang when trying to read a “MODE SENSE (6)” response from a misbehaving drive. It kept reading after the short packet. The next packet was the CSW, which it thought contained more mode sense data, and then it hung waiting for the rest of the mode sense data to arrive. I fixed the library to look for short packets and terminate the transfer early.

Because of this situation above, you might not realize a well-behaved drive has correctly stalled the endpoint until you try reading the CSW. So if you notice a stall condition after attempting to read back a CSW, clear the stall and try again. Note that the specification specifically mentions that retrying a CSW read after a stall is allowed. I suspect they allow it because of situations like the one I just described.

Safely ejecting

Some drives I tested didn’t necessarily finish saving changes before they were unplugged. For example, if I wrote data to a temporary file, and then renamed the file as my last write operation, plugging the drive into the computer showed that the rename operation hadn’t completed successfully, even though I had definitely asked the filesystem library to unmount the filesystem.

This was caused because the drive in question (a Samsung flash drive) implements caching, and I hadn’t told the drive to flush its cache to disk.

There is probably a complicated way to set up the cache properly. I believe one of the mode pages optionally returned by the “MODE SENSE (6)” command contains cache information which you might be able to configure. I came up with a simpler solution that seems to work. I send a “SYNCHRONIZE CACHE (10)” command after I’m done and have told the filesystem library to unmount the filesystem. This fixed the issue with the Samsung drive. Some drives don’t support this command and return an error, but it doesn’t seem to hurt anything.

Unsupported commands

I have read online that some poorly-designed flash drives will stop responding if you send them a command they don’t support. I haven’t observed any such drives in action. If you are concerned about this, you might want to consider removing the “PREVENT ALLOW MEDIUM REMOVAL” command from the initialization sequence, because I’ve seen many drives that don’t support it. You might also want to find a safer way of flushing the write cache than what I chose. In my use case, the very last command I send prior to the drive being ejected is the “SYNCHRONIZE CACHE” command. If that command is unsupported and causes the drive firmware to crash, the drive probably doesn’t support caching anyway, so I can assume the data is already safely written to the disk and the user is about to unplug it anyway.

Final thoughts

If at all possible, get a hardware USB analyzer. They’re typically expensive, but they give you so much detail about everything that’s going on. I was lucky enough to find one on eBay for a reasonable price. I can’t imagine that I would have been able to do this level of troubleshooting without one.

Even if you don’t have a hardware analyzer, if you follow some of my suggestions above, your device will be much more likely to be compatible with all of the random flash drives your end users happen to try out.

For 100% compatibility, it would probably be best to try to replicate exactly what Windows does when a drive is plugged in. However, doing that is difficult. It’s sometimes hard to understand why Windows sends the commands it does.