Android File Storage System and Adaptable Storage in Marshmallow Explained

By -

First of all, I want to say at the outset that the Android File storage system is a tragedy. It’s a miracle it somehow still works today. There are too many partitions, too many folders, and too many locations to store files. Android doesn’t even break down your used storage properly – my used space + my free space never adds up to the total available space. There have been more breaking API changes in the Android SDK regarding storage than I can keep track of.

Naturally, there is a lot of confusion regarding storage among Android users. With adoptable storage to add to the confusion, some of the things people say are downright wrong. I wanted to make a post that presents the facts once and for all. I’ll explain in terms of how it works on my Moto G 2015 (XT1550). It has a real SD card that I’ve been using for adoptable storage for the past two months. I’ve rooted my device for exploration purposes, but I’ll limit my explanation to how it would behave on a stock device. I’m reasonably confident in the overall picture I’ve gathered, but I realize my knowledge incomplete, so I welcome any corrections.

Also checkout: Data Privacy Tips for Android Users.

Android File Storage System

My phone has 16GB of storage. I suspect this actually means 16 billion bytes of storage, which translates to only 14.9 GB since the OS uses 1024 for the conversions (1024 bytes = 1 KB, 1024 KB = 1 MB and so on…). Henceforth, I will refer to storage when calculated based on 1024 as MiB and GiB, etc.

Of the 14.9 GiB, only 11.92 GB is available to the user, though. This is because the overall storage of the device is sectioned off into many partitions (there are 42!!!), only one of which you, the user, directly has access to (the /data one). I’ll list some of the important partitions:

  • /system – 2.3 GiB (ext4 file system): This is the “ROM” of the device. It basically contains the Android OS and is the bulk of what’s updated when you get software updates. It is mounted as read-only and can only be officially written to by an update that is cryptographically signed by Motorola (unless you “unlock” your bootloader).
  • /cache – 256 MiB (ext4 file system): This, as the name implies, is a partition used to store temporary files. Note that this is different from cache files stored by apps, however. When you look in your storage break down in the settings screen of Android, the cache used by applications is not this partition. That lives in the /data partition. /cache is not even readable by applications. I suspect it’s used by the system for its own purposes. Would appreciate any explanations you guys can offer for this one.
  • /data – 11.9 GiB (Flash Friendly file system): The is the userdata partition. It’s what the user can mainly manipulate. It contains your user-installed apps (and the updates to the system apps) and almost all your app data. I’ll get to that almost part in a bit. Data that an app stores inside your /data is guaranteed by the OS to remain private, in the sense that no other app or the user can access it on an unmodified device (disregarding the ADB backup/restore feature for apps and data). See appendix I regarding all the stuff inside /data. The part important for this guide is that there is just one folder – /data/media/0 – that is exposed to the user and apps as the “usb storage” or “internal storage” or “internal sd card”. It’s the root of the folders we’ve come to know and love like DCIM, Music, Downloads, etc… Files stored by the system under /data that are not under /data/media are not visible to users. You can only reclaim that space by a factory reset. I’m still looking at what kinds of logs might be stored under /data and how quickly they grow, consuming precious internal storage.

Whew! They add up to roughly 14.5 GiB, just 400 MiB shy of my device’s 14.9 GiB. It doesn’t stop here, sadly. There are more partitions, such as the cell modem, fsg and more that each take a (small) slice out of my device’s 14.9 GiB total space. I’m not sure what they all do but there are a total of 42 of them! An interesting partition is /persist. While I’m not certain about this, I suppose this partition is what allows features like device protection work even across factory resets. Factory reset only wipes the /data partition, so /persist, well, persists. Users and apps, as expected, can’t access /persist on unrooted devices though. I hope this part shows users why they have less space than the advertised amount. It is not “lost due to formatting” – that is only tens of MiBs at most. It is mostly a little shady advertising carried over from the HDD days in computers and the fact that some of your device’s space is reserved for other partitions – you never even see those files.

Also checkout: 5 Little known Google Apps.

SD Cards (Portable Mode in Marshmallow)

As you may have noticed, I haven’t said a thing about SD cards yet. This is because they are treated completely differently. When I use my Moto G without a physical SD card in the device, the OS still simulates an “internal” SD card. This “internal” SD card is accessed via the /storage/emulated directory (which contains numbered directories for eac user). So a file manager is actually browsing /storage/emulated/0 when it’s looking at user storage. This directory actually is backed by data in /data/media – /storage/emulated is just a mount point. When I do put in a physical SD card, it is mounted at /storage/53D-DS32 where 53D-DS32 is just some unique identifier for the card. User-installed apps can not write any storage outside /storage/emulated/0 so when you want to capture pictures to your external SD card using a downloaded camera app, you have to “select the SD card” in an additional prompt – this is since KitKat I believe. When you use an SD card in portable mode, you essentially have two locations to store user data: /storage/emulated/0 (the internal storage) and /storage/53D-DS32 (your SD card). This is a CHORE to maintain. I hated it. You’re always wondering whether this application is storing data on my internal storage or on my SD card. And applications usually don’t even offer you the option to decide where to store user data (I’m looking at you, WhatsApp). The only reasonable way to use an SD card in portable mode on Android is just use it for Pictures and Music. If your device has limited internal storage and you want to download lots of apps, tough. Your SD card wouldn’t help you before Marshmallow (see Appendix 3 for a severely crippled version of “move app to SD” Android had before Marshmallow). Apps can no longer be moved to the SD card if you’re using it in Portable Mode in Marshmallow.

Some OEMs are allowing apps to be moved to SD in portable mode on Marshmallow. So, it depends on your device. It seems like since both Moto phones and Android One phones don’t support moving apps to SD with portable storage, this is how it is in vanilla Android. Also, files on portable storage are visible to all users on the device. Files on internal storage or adopted internal storage are private to each user.

Also Check: How to remove status bar icon from status bar in Android

Adoptable storage

When you adopt an SD card as internal storage, however, all hell breaks loose. No, I’m just kidding it’s actually quite simple. The user is promising Android that “I won’t remove this SD card from the device; please integrate it with your internal storage. I understand that if I remove the card, I don’t expect my usb storage data to be there or some apps to work properly”.

But, the physical reality is that the SD card CAN be removed and SD cards do fail prematurely sometimes. So, Google couldn’t actually just combine the SD card with your device’s internal storage. It wouldn’t work. Think about it. If the card failed or a curious user removed the card just to see what happened (that would be me), NO file under /data (your userdata paritition, remember) would be guaranteed to be available. In fact, removing the SD card could cause such file corruption, that the only solution would be wiping the filesystem under /data and starting over (factory resetting the device). Clearly, this is not a viable route. Your PHONE couldn’t even be guaranteed to be stable enough to call 911 this way if the SD card failed just at the wrong time.

The only other way to “adopt” the SD card was to keep Android file storage system separate from /data and offer the user choice of where to store what. This is exactly what Google has done. They give you an option to keep apps (everything in /data/app/package.name) and apps’ private data (/data/data/package.name) on internal storage or the adopted SD card. One thing that I don’t like is that ALL apps are not movable to the adopted SD card still. No system app can be moved, which makes sense. But even for user apps, it seems like the developer has to mark the app as “movable to SD” in the same as they used to for the Move to SD feature until Lollipop. It’s opt-in, basically, from the developer’s side. I don’t like this, because the user is already promising not to remove the SD card. They’ll understand if apps are unavailable/don’t work properly without their adopted SD card.

With the “usb storage” “user-accessible storage” (the /storage/emulated folder with DCIM, Music, etc folders) aspect of your data, Google did something reasonable. Instead of forcing the user to pick for EVERY file whether they want it stored in the device’s internal storage or the adopted storage, they simply made the internal memory unavailable for storage. That is, /storage/emulated is mounted to point to your adopted SD card and your previous internal storage data that actually lived in /data/media is moved over to its new home on the SD card. This gives the added benefit that there is NO ambiguity where apps are storing user data. It always goes to the adopted SD card. See appendix 4 for my hunch on why they encrypt the adopted card.

There’s still a little confusion on adoptable storage. I’ll add the minute details to appendix 5. They are quite complex and don’t really change the overall concept. Suffice it to say, with adoptable storage, when you move an app to SD, all of its code and data is moved to SD, unlike pre-Marshmallow. If you read appendix 5, you’ll there is one exception, though.

Appendix 1 – The various folders inside /data

You can not look at the raw files and folders of Android file storage system inside this partition without rooting your device. I’ll list some of the interesting ones:

  • /data/app – This contains all user user-installed and updated system apps. Each app gets a folder by its package name (org.mozilla.firefox). Inside the folder, there is a base.apk which is what you download. A lib folder for any libraries. There is also, curiously, on devices with ART, an “oat” file. This is the compiled version of the app. The oat file is usually as big as the apk and lib combined. However, space consumed by oat files is not counted by my Moto G when looking at the storage & usb section of settings. This is absolutely ridiculous. My apps are effectively consuming twice the space that the phone tells me they are (before factoring space consumed by app data) and I had no idea where my space was going prior to rooting it.
  • /data/data – This contains the application’s private data that you can “clear” from the app management screen. It also gets automatically delete when you uninstall the app so it’s not a good candidate for storing data to persist across app installations (think chat apps like WhatsApp). This is why apps many times choose to litter the root directory of the user storage with the crap instead of storing it in the system managed directory. There is no way for an application to know when it’s about to be uninstalled and save some of its files, so they dump them all into user storage.
  • /data/dalvik-cache – This contains the dalvik VMs byte-code that then gets cross-compiled into the oat file. Not sure why this is needed honestly on phones with ART. It contains 321 MiB worth files that seem to be a complete waste of device storage.
  • /data/tombstones – This contains the crash reports when your apps stop responding. I guess this folder will also grow over time, consuming increasing amounts of space until you reset your device. Since I just rooted my phone, I can’t be sure about this though since apps haven’t crashed yet. Somebody else may have better information about this.
  • /data/media – Ah. Here we are. This contains the files that you see when you use a file manager. You don’t even see your entire userdata partition! Just one folder in it. Actually it’s just one folder inside /data/media, typically /data/media/0. You see, Android is a multi-user operating system now. So, there have to be boundaries between how much data users can share and see. Any and all apps are shared system-wide. They are updated everywhere even if one user updates one. But, the user data is unique to every user on the device. This includes the app’s private data in /data/data and the “internal SD card” the user can see with file managers. So, the first user of the device will see /data/media/0, the second one will see /data/media/1 and so on… In fact, you don’t get to see /data/media/0 directly either (remember the user and apps have no access to /data)! It’s mounted to a different location (/storage/emulated) via the External SD Card Filesystem. This loop-back mount filesystem is what enforces application permissions such as WRITE_EXTERNAL_STORAGE from the SDK or the user-configurable “storage” permission in Marshmallow. There is one more folder in /data/media. It’s /data/media/obb. When you download huge games that download “additional data”, this is where it goes. It can not be user-specific, since it’s application code (or app resources like artwork but whatever). It’s curious to me why the obb folder lives in /data/media instead of just /data but I suspect that may be due to historical reasons since obb folder used to live on the first “SD Card” of the device (whether that was internal storage or an actual SD Card) before multi-user came along.

Appendix 2 – App data stored on SD card

Your device’s “usb storage” has a folder called ‘Android’ on it. You can see it in any file manager even with an unmodified device. This folder is meant to hold app data that developers don’t mind exposing to the world. Applications don’t need any special permissions to write to their special folder inside ‘Android’. Note that while other applications CAN read data that one application puts in its ‘Android’ folder on the SD card, it’s not meant to be used to share user files like pictures and music. App data inside the ‘Android’ folder is still deleted when the application is uninstalled. Google used this folder as a way to provide reprieve to users that had limited device memory but lots of SD card space before the adoptable storage feature in Marshmallow. The idea was that if the application didn’t care that its data could be read by other apps and the user, it could put files on the SD card instead of straining the limited device storage. This was perfect for streaming music applications, for example, that supposedly encrypted their cached music files anyway. Now that adoptable storage is here, it is my opinion that it makes little sense to store app data this way and it should be phased out in future Android versions (causing yet another breaking API change with regards to storage…). This is because the SD card is either being used in adoptable storage mode, which means the user can move the entire application and its private data to the SD card anyway. Or, the SD card is being used as Portable Storage (more akin to how you’d use an SD card in your laptop or something). This would make it a poor location to put application data on since the user might remove the card at any time. Moreover, lots of apps just dump data to their special ‘Android’ folder on the first SD card, which is usually the device’s internal storage simulated as an SD card anyway so the benefit of saving space on the device is gone. See why it’s confusing? We’re better off without it in Android N.

Appendix 3 – How the native “move apps to SD” feature worked before Marshmallow

Before adoptable storage in Marshmallow, Android allowed apps to be moved to the SD card in a limited capacity. Only if the developer marked their app as movable and only a FEW of the app’s files would be moved over. Specifically, the apps’ obb files always stayed on internal storage and they are some the most space consuming files! Moreover, apps frequently moved back to internal storage when they got updated from the Play store. This feature was more of hassle and I rarely bothered with it.

Appendix 4 – Why encrypt the SD card

Before adoptable storage, when Android moved an app to SD the way appendix 3 describes, it would create a folder under .android_secure on your SD card. The app’s files under .android_secure would be INDIVIDUALLY encrypted to maintain app privacy because an SD card can always be browsed on a computer. Remember, Android guarantees app data stored in /data/data is private to the app on an unmodified device. I guess this isn’t very good for performance. With adoptable storage, they needed to store all of an app’s files and private data on the SD card. They could have done this with just formatting it as EXT4 but remember, an SD card can always be pulled out and put into a laptop. They didn’t want to expose apps’ private data this way probably, so they chose to encrypt the entire device. See this excellent blog post on recovering the encryption key for an adopted SD card. You’ll need to be rooted, obviously.

Appendix 5 – More on adoptable storage

The layout of the SD card once it’s adopted as internal storage looks a lot like the /data partition. The adopted SD card is actually mounted at /mnt/expand/$UUID. Some folders within this directory:

  • /app: Equivalent of /data/app from appendix 1 for the apps you move to SD card. This includes the apk, libs, and the compiled oat file. Individual apps and their data, can be kept on internal storage or moved to the SD card if allowed by the developer. You’re not forced to place them all in one place. If you do choose to move an app to SD, all of it and its associated data goes to the SD card.
  • /user/0: Equivalent of /data/data from appendix 1 for apps you move to SD. So this means an app’s private data is also moved to SD.

So, you see, an app is completely moved to SD, unlike previous implementations of the app to SD feature in Android. The only exception here is (it’s complicated…) the obb files. All apps’ obb files reside in /storage/emulated/obb. So, if your /storage is pointing to the internal partition, all apps’ obb files will reside on internal storage. If your /storage is pointing to the adopted storage, all apps’ obb files will be on the adopted storage, regardless of whether the app itself is on internal storage or the SD card. While this decision to house obb files under /storage is odd to me, it shouldn’t matter much because I think the most common use case will be to shove a huge SD card in the device and use it as your /storage/emulated.

So, is it worth it? Yes! I just threw a 64 GB card in there and adopted it. You have to pry open the water-resistant back cover to access the SD card on the Moto G, anyway, so I’d never remove it. If I want to transfer data, I just use a USB cable or something like Air Droid. Now, I don’t have to worry about if I’ll run out of space while shooting a 1080p video, for example. Games can usually be moved to SD also, along with their obb files so I effectively have infinite space for apps, pictures and music. It works well. Is it slow? Not really that I can tell. Writing from /dev/random, the adopted storage benches within 1% of the internal storage anyway. The only downside, as I mentioned, is that developers still control whether their app is movable to SD. I don’t agree with this decision on an adopted storage device and hope it gets changed in the future.

Also Checkout:

This article is taken from Android Subreddit. We would also like to thank the author of this post, who doesn’t want any credit, head on to the original post to discuss it with other Android users.

If you have any question regarding Android File Storage system. Let me know in the comments.

MetalHead \nn/ | Balochistan |Uz garzam lewanay | Follow up on Twitter

3 Comments to Android File Storage System and Adaptable Storage in Marshmallow Explained

  1. Hello.

    It seems like you copied this post from a reddit post. Is this true? Did you ask the user? I see no credits.

Leave a Reply

Your email address will not be published.