[POC] PROPFIND with Depth:infinity and streaming XML

[michaelstingl]

• Core Issue: Depth infinity: Make recursive PROPFIND calls cheaper core#14531
• Core PR: [full-ci] Enable streaming for propfind core#38583
• Internal: https://github.com/owncloud/enterprise/issues/4504

@DeepDiver1975 did the server side arrive in the master branch? Link to pr/issue? Is there a capability for the clients, so they can check for such magic?

@felix-schwarz what else do you need to start a prototype implementation?

[DeepDiver1975]

did the server side arrive in the master branch?

no

Link to pr/issue?

owncloud/core#38583

Is there a capability for the clients, so they can check for such magic?

infinity depth is in the code base since day one .... it was simply breaking apart until today ;-)

but we can add a cap to tell client when to use this .....

[felix-schwarz]

@DeepDiver1975 Does it make sense to develop against owncloud/core#38583 yet?

@michaelstingl I only need a server to develop and test against (or a way to set one up locally in a Ubuntu VM or Docker).

[DeepDiver1975]

@DeepDiver1975 Does it make sense to develop against owncloud/core#38583 yet?

yes

[michaelstingl]

@michaelstingl I only need a server to develop and test against (or a way to set one up locally in a Ubuntu VM or Docker).

I'll take care.

[felix-schwarz]

@DeepDiver1975 Great!
@michaelstingl Thanks! Please ping me when something is ready.

[michaelstingl]

Okay, I performed the following steps in my docker-compose setup:

Replace stable 10.7 with daily *.tar

docker-compose exec owncloud bash

cd /var/www
mv owncloud owncloud.docker
wget https://download.owncloud.org/community/daily/owncloud-daily-master.tar.bz2
tar xjf owncloud-daily-master.tar.bz2 
rm -rf owncloud/config/
mv owncloud.docker/config owncloud/
mv owncloud.docker/custom owncloud/
occ upgrade

Apply patch

wget https://patch-diff.githubusercontent.com/raw/owncloud/core/pull/38583.patch
cd owncloud
patch -p1 < ../38583.patch

Add 3906 directories and 19530 files to user bob

cd /var/www
wget https://raw.githubusercontent.com/LLNL/fdtree/master/fdtree.bash
chmod +x fdtree.bash
bash fdtree.bash -f 5 -d 5 -l 5 -C
mv LEVEL0* /mnt/data/files/bob/files/
occ files:scan bob

[michaelstingl]

I saw data streaming from the first second:

% curl -XPROPFIND --user "bob" https://prop-infinity.shniq.cloud/remote.php/webdav -H "Depth:infinity" | xmllint --format - | grep href | wc -l
Enter host password for user 'bob':
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.0M    0 13.0M    0     0   760k      0 --:--:--  0:00:17 --:--:--  765k
   23444

23444 items in 17 seconds doesn't sound too bad, and this was a tiny CPX11 Wölkchen from Hetzner (2vCPU, 2GB RAM, 40GB Disk space on local NVMe SSD)

[michaelstingl]

Detailed/full instructions for local Docker setup:

Start local 10.7 Docker

docker run --rm -d \
  --name owncloud \
  -p 18080:8080 \
  -e OWNCLOUD_APPS_ENABLE=oauth2 \
  -e ADMIN_USERNAME=admin \
  -e ADMIN_PASSWORD=admin \
  owncloud/server:10.7

Replace stable 10.7 with daily *.tar

docker exec -ti owncloud bash

cd /var/www
mv owncloud owncloud.docker
wget https://download.owncloud.org/community/daily/owncloud-daily-master.tar.bz2
tar xjf owncloud-daily-master.tar.bz2 
rm -rf owncloud/config/
mv owncloud.docker/config owncloud/
mv owncloud.docker/custom owncloud/
occ upgrade

Apply patch

wget https://patch-diff.githubusercontent.com/raw/owncloud/core/pull/38583.patch
cd owncloud
patch -p1 < ../38583.patch

Create user bob & connect to create data dir

curl -X POST http://admin:admin@localhost:8080/ocs/v1.php/cloud/users -d userid="bob" -d password="password"
curl -XPROPFIND --user "bob:password" "http://localhost:8080/remote.php/webdav"

Add 3906 directories and 19530 files to user bob

cd /var/www
wget https://raw.githubusercontent.com/LLNL/fdtree/master/fdtree.bash
chmod +x fdtree.bash
bash fdtree.bash -f 5 -d 5 -l 5 -C
mv LEVEL0* /mnt/data/files/bob/files/
occ files:scan bob

Try PROPFIND with Depth:Infinity

curl -XPROPFIND --user "bob:password" "http://localhost:8080/remote.php/webdav" -H "Depth:infinity" -o "propfind.xml"

Result:

root@00340d875dfe: /var/www # curl -XPROPFIND --user "bob:password" "http://localhost:8080/remote.php/webdav" -H "Depth:infinity" -o "propfind.xml"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12.9M    0 12.9M    0     0  2910k      0 --:--:--  0:00:04 --:--:-- 2980k

Result with new WebDAV endpoint:

root@00340d875dfe: /var/www # curl -XPROPFIND --user "bob:password" "http://localhost:8080/remote.php/dav/files/bob" -H "Depth:infinity" -o "propfind2.xml"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.1M    0 13.1M    0     0  2893k      0 --:--:--  0:00:04 --:--:-- 2973k

[michaelstingl]

I thought about the following error cases:

• Auth token expiration faster than XML download
• Invalid XML somewhere in the middle of the response
  (ignore everything after XML-error for now, maybe log "good XML before error size" and "full response size")
• Plenty of valid XML, but response ends with HTTP error

[DeepDiver1975]

With respect to error handling this needs to be deeper worked on.

In the non streaming case:

• we walk the file tree
• if errors occur -> exception is thrown
• exception is converted to http error status
• resp body holds error xml

Current streaming case:

• http response status code 207 is sent
• we walk the file tree and start writing xml response
• if error occurs -> exception is thrown
• we stop processing the file tree -> xml response is unfinished and at least partially invalid
• no error xml is sent to the client

Possible solution I:

• do a pre-processing step to catch potential errors
• in case of errors -> http error status and xml is sent to client
• if successful -> stream xml
• PROBLEM: the preprocessing step might require to walk the full file tree - this is time consuming and I fear we loose all the performance gain

Possible solution II:

• just like the current implementation
• in case of errors no exception is throws which terminates the tree walk but an xml entity with status 40x is written to the xml response body
• this requires in depth analysis and testing of the code base to make sure not a single exception is accidentally slipping through

[felix-schwarz]

First results testing the iOS implementation:

Test set prop-infinity.shniq.cloud

• Folders: 3908
• Files: 19535
• Total: 23443 items

Results

Method	Traffic in	Traffic out	CPU	Network time	Processing time	Total time (mm:ss)
Individual	20 MB	2 MB	200% (app+FP)	n/a	n/a	7:09
Infinite	17 MB	17 KB	99% (app)	0:15	0:18	0:33

Error handling

• the implementation keeps track of which folders have been fully received
• if parsing is cancelled by the user or an XML error is encountered:
  • already parsed items are kept
  • schedules individual scans for folders whose contents has not yet been fully received (so item metadata will be fully retrieved eventually)

Summary

• Infinite PROPFIND is 13 x faster in total compared to individual PROPFINDs (in "split" implementation)
• "parallel" implementation, where incoming PROPFIND XML data is parsed immediately, could give a total speedup of up to 23 x

User interface

The infinite PROPFIND is automatically performed after account setup, while keeping the user informed and providing an option to cancel it (through a button labeled Skip)

HTTP phase	parsing phase