← Back
Editing: PERFORMANCE.md
# Test setup Processor: x86_64, Intel(R) Xeon(R) Platinum 8369B CPU @ 2.70GHz * 2 VCores Storage: Cloud disk, 3000 IOPS upper limit OS Kernel: Linux 6.2 Software: LZ4 1.9.3, erofs-utils 1.6, squashfs-tools 4.5.1 Disclaimer: Test results could be varied from different hardware and/or data patterns. Therefore, the following results are **ONLY for reference**. # Benchmark on multiple files [Rootfs of Debian docker image](https://github.com/debuerreotype/docker-debian-artifacts/blob/dist-amd64/bullseye/rootfs.tar.xz?raw=true) is used as the dataset, which contains 7000+ files and directories. Note that that dataset can be replaced regularly, and the SHA1 of the snapshot "rootfs.tar.xz" used here is "aee9b01a530078dbef8f08521bfcabe65b244955". ## Image size | Size | Filesystem | Cluster size | Build options | |-----------|------------|--------------|----------------------------------------------------------------| | 124669952 | erofs | uncompressed | -T0 [^1] | | 124522496 | squashfs | uncompressed | -noD -noI -noX -noF -no-xattrs -all-time 0 -no-duplicates [^2] | | 73601024 | squashfs | 4096 | -b 4096 -comp lz4 -Xhc -no-xattrs -all-time 0 | | 73121792 | erofs | 4096 | -zlz4hc,12 [^3] -C4096 -Efragments -T0 | | 67162112 | squashfs | 16384 | -b 16384 -comp lz4 -Xhc -no-xattrs -all-time 0 | | 65478656 | erofs | 16384 | -zlz4hc,12 -C16384 -Efragments -T0 | | 61456384 | squashfs | 65536 | -b 65536 -comp lz4 -Xhc -no-xattrs -all-time 0 | | 59834368 | erofs | 65536 | -zlz4hc,12 -C65536 -Efragments -T0 | | 59150336 | squashfs | 131072 | -b 131072 -comp lz4 -Xhc -no-xattrs -all-time 0 | | 58515456 | erofs | 131072 | -zlz4hc,12 -C131072 -Efragments -T0 | [^1]: Forcely reset all timestamps to match squashfs on-disk basic inodes for now. [^2]: Currently erofs-utils doesn't actively de-duplicate identical files although the on-disk format supports this. [^3]: Because squashfs uses level 12 for LZ4HC by default. ## Sequential data access ```bash hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "tar cf - . | cat > /dev/null" ``` | Filesystem | Cluster size | Time | |------------|--------------|---------------------------------| | squashfs | 4096 | 10.257 s ± 0.031 s | | erofs | uncompressed | 1.111 s ± 0.022 s | | squashfs | uncompressed | 1.034 s ± 0.020 s | | squashfs | 131072 | 941.3 ms ± 7.5 ms | | erofs | 4096 | 848.1 ms ± 17.8 ms | | erofs | 131072 | 724.2 ms ± 11.0 ms | ## Sequential metadata access ```bash hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "tar cf /dev/null ." ``` | Filesystem | Cluster size | Time | |------------|--------------|---------------------------------| | erofs | uncompressed | 419.6 ms ± 8.2 ms | | squashfs | 4096 | 142.5 ms ± 5.4 ms | | squashfs | uncompressed | 129.2 ms ± 3.9 ms | | squashfs | 131072 | 125.4 ms ± 4.0 ms | | erofs | 4096 | 75.5 ms ± 3.5 ms | | erofs | 131072 | 65.8 ms ± 3.6 ms | [ Note that erofs-utils currently doesn't perform quite well for such cases due to metadata arrangement when building. It will be fixed in the later versions. ] ## Small random data access (~7%) ```bash find mnt -type f -printf "%p\n" | sort -R | head -n 500 > list.txt hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "cat list.txt | xargs cat > /dev/null" ``` | Filesystem | Cluster size | Time | |------------|--------------|---------------------------------| | squashfs | 4096 | 1.386 s ± 0.032 s | | squashfs | uncompressed | 1.083 s ± 0.044 s | | squashfs | 131072 | 1.067 s ± 0.046 s | | erofs | 4096 | 249.6 ms ± 6.5 ms | | erofs | uncompressed | 237.8 ms ± 6.3 ms | | erofs | 131072 | 189.6 ms ± 7.8 ms | ## Small random metadata access (~7%) ```bash find mnt -type f -printf "%p\n" | sort -R | head -n 500 > list.txt hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "cat list.txt | xargs stat" ``` | Filesystem | Cluster size | Time | |------------|--------------|---------------------------------| | squashfs | 4096 | 817.0 ms ± 34.5 ms | | squashfs | 131072 | 801.0 ms ± 40.1 ms | | squashfs | uncompressed | 741.3 ms ± 18.2 ms | | erofs | uncompressed | 197.8 ms ± 4.1 ms | | erofs | 4096 | 63.1 ms ± 2.0 ms | | erofs | 131072 | 60.7 ms ± 3.6 ms | ## Full random data access (~100%) ```bash find mnt -type f -printf "%p\n" | sort -R > list.txt hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "cat list.txt | xargs cat > /dev/null" ``` | Filesystem | Cluster size | Time | |------------|--------------|---------------------------------| | squashfs | 4096 | 20.668 s ± 0.040 s | | squashfs | uncompressed | 12.543 s ± 0.041 s | | squashfs | 131072 | 11.753 s ± 0.412 s | | erofs | uncompressed | 1.493 s ± 0.023 s | | erofs | 4096 | 1.223 s ± 0.013 s | | erofs | 131072 | 598.2 ms ± 6.6 ms | ## Full random metadata access (~100%) ```bash find mnt -type f -printf "%p\n" | sort -R > list.txt hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "cat list.txt | xargs stat" ``` | Filesystem | Cluster size | Time | |------------|--------------|---------------------------------| | squashfs | 131072 | 9.212 s ± 0.467 s | | squashfs | 4096 | 8.905 s ± 0.147 s | | squashfs | uncompressed | 7.961 s ± 0.045 s | | erofs | 4096 | 661.2 ms ± 14.9 ms | | erofs | uncompressed | 125.8 ms ± 6.6 ms | | erofs | 131072 | 119.6 ms ± 5.5 ms | # FIO benchmark on a single large file `silesia.tar` (203M) is used to benchmark, which could be generated from unzipping [silesia.zip](http://mattmahoney.net/dc/silesia.zip) and tar. ## Image size | Size | Filesystem | Cluster size | Build options | |-----------|------------|--------------|-----------------------------------------------------------| | 114339840 | squashfs | 4096 | -b 4096 -comp lz4 -Xhc -no-xattrs | | 104972288 | erofs | 4096 | -zlz4hc,12 -C4096 | | 98033664 | squashfs | 16384 | -b 16384 -comp lz4 -Xhc -no-xattrs | | 89571328 | erofs | 16384 | -zlz4hc,12 -C16384 | | 85143552 | squashfs | 65536 | -b 65536 -comp lz4 -Xhc -no-xattrs | | 81211392 | squashfs | 131072 | -b 131072 -comp lz4 -Xhc -no-xattrs | | 80519168 | erofs | 65536 | -zlz4hc,12 -C65536 | | 78888960 | erofs | 131072 | -zlz4hc,12 -C131072 | ## Sequential I/Os ```bash fio -filename=silesia.tar -bs=4k -rw=read -name=job1 ``` | Filesystem | Cluster size | Bandwidth | |------------|--------------|-----------| | erofs | 65536 | 624 MiB/s | | erofs | 16384 | 600 MiB/s | | erofs | 4096 | 569 MiB/s | | erofs | 131072 | 535 MiB/s | | squashfs | 131072 | 236 MiB/s | | squashfs | 65536 | 157 MiB/s | | squashfs | 16384 | 55.2MiB/s | | squashfs | 4096 | 12.5MiB/s | ## Full Random I/Os ```bash fio -filename=silesia.tar -bs=4k -rw=randread -name=job1 ``` | Filesystem | Cluster size | Bandwidth | |------------|--------------|-----------| | erofs | 131072 | 242 MiB/s | | squashfs | 131072 | 232 MiB/s | | erofs | 65536 | 198 MiB/s | | squashfs | 65536 | 150 MiB/s | | erofs | 16384 | 96.4MiB/s | | squashfs | 16384 | 49.5MiB/s | | erofs | 4096 | 33.7MiB/s | | squashfs | 4096 | 6817KiB/s | ## Small Random I/Os (~5%) ```bash fio -filename=silesia.tar -bs=4k -rw=randread --io_size=10m -name=job1 ``` | Filesystem | Cluster size | Bandwidth | |------------|--------------|-----------| | erofs | 131072 | 19.2MiB/s | | erofs | 65536 | 16.9MiB/s | | squashfs | 131072 | 15.1MiB/s | | erofs | 16384 | 14.7MiB/s | | squashfs | 65536 | 13.8MiB/s | | erofs | 4096 | 13.0MiB/s | | squashfs | 16384 | 11.7MiB/s | | squashfs | 4096 | 4376KiB/s |
Save File
Cancel