First, I want to say at the outset that the Android File storage system is a tragedy. It’s a miracle it still works today. There are too many partitions, too many folders, and too many locations to store files. Android doesn’t even break down your used storage properly—my used space + my free space never adds up to the total available space. There have been more breaking API changes in the Android SDK regarding storage than I can keep track of.
Naturally, there is a lot of confusion regarding storage among Android users. With adoptable storage to add to the confusion, some things people say are wrong. I wanted to make a post that presents the facts once and for all. I’ll explain how it works on my Moto G 2015 (XT1550). It has an SD card I’ve used for adoptable storage for the past two months. I’ve rooted my device for exploration purposes, but I’ll limit my explanation to how it would behave on a stock device. I’m reasonably confident in the overall picture I’ve gathered, but I realize my knowledge is incomplete, so I welcome any corrections.
Also checkout: Data Privacy Tips for Android Users.
Android File Storage System
My phone has 16GB of storage. I suspect this means 16 billion bytes of storage, which translates to only 14.9 GB since the OS uses 1024 for the conversions (1024 bytes = 1 KB, 1024 KB = 1 MB and so on…). Henceforth, I will refer to storage when calculated based on 1024 as MiB and GiB, etc.
Of the 14.9 GiB, only 11.92 GB is available to the user. This is because the overall storage of the device is sectioned off into many partitions (there are 42!!!), only one of which you, the user, directly have access to (the /data one). I’ll list some of the essential partitions:
- /system – 2.3 GiB (ext4 file system): This is the “ROM” of the device. It contains the Android OS and is the bulk of what’s updated when you get software updates. It is mounted as read-only and can only be officially written to by a cryptographically signed update by Motorola (unless you “unlock” your bootloader).
- /cache – 256 MiB (ext4 file system): As the name implies, this partition is used to store temporary files. Note that this is different from cache files stored by apps. When you look at your storage breakdown in the Android settings screen, the cache used by applications is not this partition. That lives in the /data partition. /cache is not even readable by applications. I suspect the system uses it for its purposes. I would appreciate any explanations you guys can offer for this one.
- /data – 11.9 GiB (Flash Friendly file system): This user data partition. It’s what the user can mainly manipulate. It contains your user-installed apps (and the updates to the system apps) and almost all your app data. I’ll get to that nearly part in a bit. Data that an app stores inside your /data is guaranteed by the OS to remain private, in the sense that no other app or the user can access it on an unmodified device (disregarding the ADB backup/restore feature for apps and data). See Appendix I regarding all the stuff inside /data. The critical part of this guide is that there is just one folder – /data/media/0 – that is exposed to the user and apps as the “USB storage”, “internal storage”, or “internal SD card”. It’s the root of the folders we’ve come to know and love, like DCIM, Music, Downloads, etc… Files stored by the system under /data that are not under /data/media are not visible to users. You can only reclaim that space with a factory reset. I’m still looking at what logs might be stored under /data and how quickly they grow, consuming precious internal storage.
Whew! They add up to roughly 14.5 GiB, just 400 MiB, shy of my device’s 14.9 GiB. It doesn’t stop here, sadly. There are more partitions, such as the cell modem, fig and more,s, each taking a (small) slice out of my device’s 14.9 GiB total space. I’m not sure what they all do, but there are 42 of them! An exciting partition is /persists. While I’m not sure about this, I suppose this partition allows features like device protection to work even across factory resets. Factory reset only wipes the /data partition, so /persist, well, persists. Users and apps, as expected, can’t access /persist on unrooted devices. I hope this part shows users why they have less space than advertised. It is not “lost due to formatting” – only tens of MiBs at most. It is primarily shady advertising carried over from the HDD days in computers and the fact that some of your device’s space is reserved for other partitions – you never even see those files.
Also checkout: 5 Little known Google Apps.
SD Cards (Portable Mode in Marshmallow)
As you may have noticed, I haven’t said a thing about SD cards yet. This is because they are treated entirely differently. When I use my Moto G without a physical SD card, the OS still simulates an “internal” SD card. This “internal” SD card is accessed via the /storage/emulated directory (which contains numbered directories for each user). So, a file manager is browsing /storage/emulated/0 when looking at user storage. This directory is backed by data in /data/media – /storage/emulated, which is just a mount point. When I put in a physical SD card, it is mounted at /storage/53D-DS32, where 53D-DS32 is just some unique identifier. User-installed apps can not write any storage outside /storage/emulated/0, so when you want to capture pictures to your external SD card using a downloaded camera app, you have to “select the SD card” in an additional prompt – this is since KitKat, I believe. When you use an SD card in portable mode, you essentially have two locations to store user data: /storage/emulated/0 (the internal storage) and /storage/53D-DS32 (your SD card). This is a CHORE to maintain. I wouldn’t say I liked it. You’re always wondering whether this application stores data on my internal storage or my SD card. And applications usually don’t even offer you the option to decide where to store user data (I’m looking at you, WhatsApp). The only reasonable way to use an Android SD card in portable mode is for Pictures and Music. If your device has limited internal storage and you want to download lots of apps, it’s tough. Your SD card wouldn’t help you before Marshmallow (see Appendix 3 for a severely crippled version of “move app to SD” Android had before Marshmallow). Apps can no longer be moved to the SD card if used in Portable Mode in Marshmallow.
Some OEMs allow apps to be moved to SD in portable mode on Marshmallow. So, it depends on your device. Since both Moto phones and Android One phones don’t support moving apps to SD with portable storage, this is how it is in vanilla Android. Also, files on portable storage are visible to all users on the device. Files on internal storage or adopted internal storage are private to each user.
Also Check: How to remove status bar icon from status bar in Android
Adaptable storage
However, when you adopt an SD card as internal storage, all hell breaks loose. No, I’m just kidding; it’s actually quite simple. The user promises Android that “I won’t remove this SD card from the device; please integrate it with your internal storage. I understand that if I remove the card, I don’t expect my USB storage data to be there or some apps to work properly.”
However, the physical reality is that the SD card can be removed, and SD cards fail prematurely sometimes. So, Google couldn’t combine the SD card with your device’s internal storage. It wouldn’t work. Think about it. If the card failed or a curious user removed the card to see what happened (that would be me), NO file under /data (your user data partition, remember) would be guaranteed to be available. Removing the SD card could cause file corruption, so the only solution would be wiping the filesystem under /data and starting over (factory resetting the device). This is not a viable route. Your PHONE couldn’t even be guaranteed to be stable enough to call 911 this way if the SD card failed just at the wrong time.
The only other way to “adopt” the SD card was to keep the Android file storage system separate from /data and offer the user a choice of where to store what. This is precisely what Google has done. They allow you to keep apps (everything in /data/app/package.name) and apps’ private data (/data/data/package.name) on internal storage or the adopted SD card. I wouldn’t say I like that ALL apps are still not movable to the adopted SD card. No system app can be moved, which makes sense. But even for user apps, it seems like the developer has to mark the app as “movable to SD” like they used to for the Move to SD feature until Lollipop. It’s opt-in, basically, from the developer’s side. I wouldn’t say I like this because the user has promised not to remove the SD card. They’ll understand if apps are unavailable/don’t work properly without their adopted SD card.
With the “USB storage” and “user-accessible storage” (the /storage/emulated folder with DCIM, Music, etc folders) aspect of your data, Google did something reasonable. Instead of forcing the user to pick EVERY file, whether they want it stored in the device’s internal storage or the adopted storage, they made the internal memory unavailable for storage. That is, /storage/emulated is mounted to point to your adopted SD card, and your previous internal storage data that lived in /data/media is moved over to its new home on the SD card. This adds the benefit that there is NO ambiguity when apps are storing user data. It always goes to the adopted SD card. See Appendix 4 for my hunch on why they encrypt the adopted card.
There’s still a little confusion about adoptable storage. I’ll add the minute details to Appendix 5. They are quite complex and don’t really change the overall concept. Suffice it to say that with adaptable storage, when you move an app to SD, all of its code and data are moved to SD, unlike pre-Marshmallow. If you read Appendix 5, you’ll see there is one exception, though.
Appendix 1 – The various folders inside /data
You can not look at the raw files and folders of the Android file storage system inside this partition without rooting your device. I’ll list some of the interesting ones:
- /data/app – This contains all user-installed and updated system apps. Each app gets a folder by its package name (org.mozilla.firefox). Inside the folder, there is a base.apk, which is what you download. A lib folder for any libraries. There is also an “oat” file on devices with ART. This is the compiled version of the app. The oat file is usually as big as the apk and lib combined. However, my Moto G does not count space consumed by oat files when looking at the storage & usb section of settings. This is ridiculous. My apps effectively consume twice the space that the phone tells me they are (before factoring in space consumed by app data), and I had no idea where my space was going before rooting it.
- /data/data – This contains the application’s private data that you can “clear” from the app management screen. It also gets automatically deleted when you uninstall the app, so it’s not a good candidate for storing data to persist across app installations (think chat apps like WhatsApp). This is why apps many times choose to litter the root directory of the user storage with the crap instead of storing it in the system-managed directory. There is no way for an application to know when it’s about to be uninstalled and save some of its files, so they dump them all into user storage.
- /data/Dalvik-cache – This contains the Dalvik VMs byte-code that then gets cross-compiled into the oat file. Honestly, I’m not sure why this is needed on phones with ART. It includes 321 MiB files that seem a complete waste of device storage.
- /data/tombstones – This contains the crash reports when your apps stop responding. This folder will also grow, increasing space until you reset your device. Since I just rooted my phone, I can’t be sure about this since apps haven’t crashed yet. Somebody else may have better information about this.
- /data/media – Ah. Here we are. This contains the files that you see when you use a file manager. You don’t even see your entire user data partition! Just one folder in it. It’s just one folder inside /data/media, typically /data/media/0. You see, Android is now a multi-user operating system. So, there must be boundaries between how much data users can share and see. Any apps are shared system-wide. They are updated everywhere, even if one user updates one. But, the user data is unique to every user on the device. This includes the app’s private data in /data/data and the “internal SD card” the user can see with file managers. So, the first user of the device will see /data/media/0, the second one will see /data/media/1 and so on… You don’t get to see /data/media/0 directly either (remember, the user and apps have no access to /data)! It’s mounted to a different location (/storage/emulated) via the External SD Card Filesystem. This loop-back mount filesystem enforces application permissions such as WRITE_EXTERNAL_STORAGE from the SDK or the user-configurable “storage” permission in Marshmallow. There is one more folder in /data/media. It’s /data/media/obb. When you download massive games that download “additional data”, this is where it goes. It can not be user-specific since it’s application code (or app resources like artwork, but whatever). It’s curious why the OBB folder lives in /data/media instead of just /data, Still, I suspect that may be due to historical reasons since the Obb folder used to live on the first “SD Card” of the device (whether internal storage or an actual SD Card) before multi-user came along.
Appendix 2 – App data stored on SD card
Your device’s “USB storage” has a folder called ‘Android’ on it. You can see it in any file manager, even with an unmodified device. This folder holds app data that developers don’t mind exposing. Applications don’t need special permissions to write to their particular folder inside ‘Android’. Note that while other applications CAN read data that one application puts in its ‘Android’ folder on the SD card, it’s not meant to share user files like pictures and music. App data inside the ‘Android’ folder is still deleted when the application is uninstalled. Google used this folder to provide a reprieve to users with limited device memory but lots of SD card space before the adaptable storage feature in Marshmallow. The idea was that if the application didn’t care that other apps and the user could read its data, it could put files on the SD card instead of straining the limited device storage. This was perfect for streaming music applications, for example, that supposedly encrypted their cached music files anyway. Now that adoptable storage is here, I think it makes little sense to store app data this way, and it should be phased out in future Android versions (causing yet another breaking API change regarding storage). s is because the SD card is used in adoptable storage mode, which means the user can move the entire application and its private data to the SD card anyway. Or, the SD card is being used as Portable Storage (more akin to how you’d use an SD card in your laptop or something). This would make it a poor location for application data. The user might remove the card at a time.
Moreover, many apps dump data into their particular ‘Android’ folder on the first SD card, which is usually the device’s internal storage simulated as an SD card anyway, so the benefit of saving space on the device is gone. See why it’s confusing? We’re better off without it in Android N.
Appendix 3 – How the native “move apps to SD” feature worked before Marshmallow
Before adoptable storage in Marshmallow, Android allowed apps to be moved to the SD card in a limited capacity. Only a FEW of the app’s files would be moved over if the developer marked their app as movable. Specifically, the apps’ OBB files always stay in internal storage and are some of the most space-consuming files! Moreover, apps frequently moved back to internal storage when updated from the Play Store. This feature was more of a hassle, and I rarely bothered with it.
Appendix 4 – Why encrypt the SD card
Before adoptable storage, when Android moved an app to SD the way Appendix 3 describes, it would create a folder under .android_secure on your SD card. The app’s files under .android_secure would be INDIVIDUALLY encrypted to maintain app privacy because an SD card can always be browsed on a computer. Remember, Android guarantees app data stored in /data/data is private to the app on an unmodified device. I guess this isn’t very good for performance. With adaptable storage, they needed to store an app’s files and private data on the SD card. They could have done this by formatting it as EXT4, but remember, an SD card can always be pulled out and put into a laptop. They didn’t want to expose the apps’ private data this way, probably, so they chose to encrypt the entire device. See this excellent blog post on recovering the encryption key for an adopted SD card. You’ll need to be rooted.
Appendix 5 – More on adoptable storage
Once it’s adopted as internal storage, the SD card layout looks a lot like the /data partition. The adopted SD card is mounted at /mnt/expand/$UUID. Some folders within this directory:
- /app: Equivalent of /data/app from appendix 1 for the apps you move to SD card. This includes the apk, libs, and the compiled OAT file. Individual apps and their data can be kept in internal storage or moved to the SD card if allowed by the developer. You’re not forced to place them all in one place. If you move an app to SD, all of it and its associated data go to the SD card.
- /user/0: The equivalent of /data/data from Appendix 1 for apps you move to SD. This means an app’s private data is also moved to SD.
So, you see, an app is completely moved to SD, unlike previous implementations of the app-to-SD feature in Android. The only exception here is (it isn’t straightforward…) the OBB files. All apps’ obb files reside in /storage/emulated/obb. So, if your /storage points to the internal partition, all apps’ obb files will reside in the internal storage. If your /storage points to the adopted storage, all apps’ OBB files will be on the adopted storage, regardless of whether the app is on internal storage or the SD card. While this decision to house OBB files under /storage is odd to me, it shouldn’t matter much because I think the most common use case will be to shove a giant SD card in the device and use it as your /storage/emulated.
So, is it worth it? Yes! I just threw a 64 GB card in there and adopted it. You have to pry open the water-resistant back cover to access the SD card on the Moto G, anyway, so I’d never remove it. If I want to transfer data, I use a USB cable or Air Droid. Now, I don’t have to worry about if I’ll run out of space while shooting a 1080p video, for example. Games can usually be moved to SD, along with their OBB files, so I have infinite space for apps, pictures, and music. It works well. Is it slow? Not really that I can tell. Writing from /dev/random, the adopted storage benches are within 1% of the internal storage. As I mentioned, the only downside is that developers still control whether their app is movable to SD. I’m afraid I have to disagree with this decision on an adopted storage device and hope it will change.
Also Checkout:
- Top 6 Lock Screen Apps.
- 3 Android Apps I can’t live without.
- Top 5 Android Launchers.
- HTC 10 QHD Stock Wallpapers.
- Five great apps that didn’t make it to Google Play Store.
This article is taken from the Android Subreddit. We would also like to thank the author of this post, who doesn’t want any credit, and head to the original post to discuss it with other Android users.
If you have any questions regarding the Android File Storage system, Let me know in the comments.
