Android 12 scoped storage hell

[hrydgard]

Android Q will introduce "scoped storage", which imposes really heavy restrictions on external storage. And in the version after Q, it will be a requirement for all apps, even those who target Android-28 (which PPSSPP will do as long as possible). So we will be able to limp along for a while longer without it, but next year, new devices will start requiring it, and it'll greatly worsen the user experience. We need to find ways to make it as workable as possible.

For us, the filesystem really is part of the UX to match the real PSP with an USB connection.

Users will additionally have problems if they install the app on an older OS, upgrade to Q, and then downgrade to an older version of the app (which is done sometimes to bisect bugs, etc).

The one bright point is that the scoped storage APIs allows getting file descriptors so I/O performance will at least not be hampered much - assuming the file descriptors are unrestricted with regards to seek etc.

https://android-developers.googleblog.com/2019/04/android-q-scoped-storage-best-practices.html

EDIT: I have requested a reprieve through AndroidManifest.xml, the hammer drops with Android 12.

[unknownbrackets]

It seems there is a way to delay it even when targeting Q:
https://developer.android.com/preview/privacy/scoped-storage#opt-out-of-filtered-view

But presumably not for R.

-[Unknown]

[nic0lette]

Would PPSSPP be able to do something like, on API levels up to 28 (or 29 with the flag @unknownbrackets references above) use the default structure.

For 29+ with scoped storage, use ACTION_OPEN_DOCUMENT_TREE and ask the user to pick the "PSP" folder.
Set the flags in the Intent to: Intent.FLAG_GRANT_READ_URI_PERMISSION | Intent.FLAG_GRANT_WRITE_URI_PERMISSION | Intent.FLAG_GRANT_PERSISTABLE_URI_PERMISSION

This will allow PPSSPP to, via ContentResolver, read and write to the folder structure there, albeit not as easily as with the File APIs.

I recommend looking at DocumentFile though I'd recommend against using it directly. Its performance is not very great =(

The other folders, including MUSIC, PICTURE, and VIDEO seem like they could be replaced with using MediaStore when inside scoped storage, or you could use a similar method to the "PSP" directory to get persistent access to those as well.

I'm guessing you already know about ParcelFileDescriptor and ParcelFileDescriptor#detachFd(), which should hopefully work for accessing files via the Storage Access Framework from C++.

Hopefully this is a bit of help.

[unknownbrackets]

Well, it seems more complicated than that. Feel free to correct me if I'm misunderstanding.

For example, PSP games will try to call sceIoLseek on returned PSP file handles. They may do something like this:

1. Open a save data file.
2. Seek to the data position (0x100) and read data.
3. Seek to the header (0x000) and read the time information.
4. Seek back to the footer (0xF00) and read more data.

Those are just example offsets, but we can't deal with the concept that we "might" get a file descriptor that doesn't support seeking. So the experience probably becomes:

• User selects a document tree.
• PPSSPP tries to open a test file in that tree and determines it's unusable.
• PPSSPP says something the user will ultimately read as "guess again, idiot."
• User gets frustrated and quits app. It doesn't work.

Unless I'm missing some way to filter the "trees", this will at best be a terrible experience, and at worst be straight up unusable on some devices.

The alternative is to try to hack around the limitations in the document API. For example, if a file is opened for read and write, read the whole thing in, then open it again for writing and wait for the game to write (possibly with seeking.) There will be tons of bugs, save data corruption, and other issues. Sounds like some kind of nightmare.

As long as we get a real file descriptor, then we should be fine I guess for saving. It's the path to get there that's the problem.

Given all those problems, I assume we won't be making users select the PSP folder by default, and instead use the hidden away app-private folder. Then we need to handle "installing" gamefaqs zips of saves, and probably need a way to transfer data in out. Hopefully one that doesn't make PPSSPP require network for you to play a single-player game that autosaves. See #1019.

And that's just about saves. Haven't even gone into how people will transfer ISOs onto their device, or navigate to homebrew. It has many of the same problems, although at least ISOs are read only (they absolutely require seeking, though.)

So - yes. It can be worked around. As long as everything is device-local storage, it's not even a big technical problem. It's just going to be very confusing for anyone who doesn't understand storage really well.

-[Unknown]

[hrydgard]

Alright, so some news!

https://developer.android.com/preview/privacy/storage has been updated with the changes for Android 11.

We have a reprieve, in that as long as we only target Android 10, we can still opt out! So that pushes the issue to the day when Google Play will no longer accept apps targeting 10, or I guess when 12 comes out. So we can probably stumble along another year or so without changes.

There's also a new MANAGE_EXTERNAL_STORAGE permission, that will be very cumbersome to explain to users how to enable though, which makes things easier for us if enabled. There's also support to use raw paths to read "external media libraries" whatever that is, might possible be useful?

Either way, no longer an emergency but I should really start preparing :P

[unknownbrackets]

Oops, this wasn't meant to be closed by that.

-[Unknown]

[hrydgard]

Bumping this to the next version, since we still have quite some time left...