Jump to content

Zstd: Difference between revisions

From Wikipedia, the free encyclopedia
Browse history interactively
← Previous edit
Content deleted Content added
VisualWikitext
Citation bot (talk | contribs)
Bots
4,985,677 edits
Add: website, title. Changed bare reference to CS1/2. | Use this bot. Report bugs. | Suggested by BrownHairedGirl | #UCB_webform 1845/1850
→‎Features: minor syntax correction
 
(47 intermediate revisions by 31 users not shown)
Line 1: Line 1:
{{short description|Lossless compression algorithm}}
{{short description|Lossless compression algorithm}}
⚫
{{lowercase}}
{{redirect|.zst|the game console emulator also using this file extension|ZSNES}}
{{redirect|.zst|the game console emulator also using this file extension|ZSNES}}
⚫
{{lowercase title}}
{{Use dmy dates|date=January 2021}}
{{Use dmy dates|date=January 2021}}
{{primary sources|date=September 2017}}
{{Infobox software
{{Infobox software
| name = Zstandard
| name = Zstandard
| author = Yann Collet
| author = Yann Collet
| developer = Yann Collet, Nick Terrell, Przemysław Skibiński<ref>{{Cite web|title=Contributors to facebook/zstd |url=https://github.com/facebook/zstd/graphs/contributors |archive-url=https://archive.today/TJKim |archive-date=27 Jan 2021 |access-date=26 Jan 2021 |website=github.com}}</ref>
| developer = Yann Collet, Nick Terrell, Przemysław Skibiński<ref>{{cite web|title=Contributors to facebook/zstd |url=https://github.com/facebook/zstd/graphs/contributors |archive-url=https://archive.today/20210127114543/https://github.com/facebook/zstd/graphs/contributors |archive-date=27 Jan 2021 |access-date=26 Jan 2021 |website=github.com}}</ref>
| operating system = [[Cross-platform]]
| operating system = [[Cross-platform]]
| genre = [[Data compression]]
| genre = [[Data compression]]
| programming language = [[C (programming language)|C]]
| programming language = [[C (programming language)|C]]
| license = [[BSD licenses#3-clause license ("BSD License 2.0", "Revised BSD License", "New BSD License", or "Modified BSD License")|BSD-3-Clause]] or [[GNU General Public License#Version 2|GPL-2.0-or-later]] (dual-licensed)
| license = Dual: [[BSD License]], GPLv2
| title =
| title =
| logo = Zstandard logo.png
| logo = Zstandard logo.png
Line 17: Line 16:
| released = {{Start date|2015|01|23|df=y}}
| released = {{Start date|2015|01|23|df=y}}
| discontinued =
| discontinued =
| latest release version = 1.5.2
| latest release version = {{wikidata|property|preferred|references|edit|P348|P548=Q2804309}}
| latest release date = {{Start date and age|2022|01|20|df=y}}<ref>{{cite web|url=https://github.com/facebook/zstd/releases|access-date=20 January 2022|title=Releases - facebook/zstd|via=[[GitHub]]}}</ref>
| latest release date = {{Start date and age|{{wikidata|qualifier|preferred|single|P348|P548=Q2804309|P577}}|df=yes}}
| latest preview version =
| latest preview version =
| platform = Portable
| platform = Portable
Line 24: Line 23:
}}
}}


'''Zstandard''', commonly known by the name of its [[reference implementation]] '''zstd''', is a [[lossless compression|lossless]] [[data compression]] algorithm developed by Yann Collet at [[Facebook]].
'''Zstandard''' is a [[lossless compression|lossless]] [[data compression]] algorithm developed by Yann Collet at [[Facebook]].
''Zstd'' is the reference implementation in [[C (programming language)|C]]. Version 1 of this implementation was released as [[open-source software]] on 31 August 2016.<ref>
'''Zstd''' is the corresponding [[reference implementation]] in [[C (programming language)|C]], released as [[open-source software]] on 31 August 2016.<ref>
{{cite news
{{cite news
|author=Sergio De Simone
|author=Sergio De Simone
Line 33: Line 32:
|access-date=2019-04-20
|access-date=2019-04-20
|date=2016-09-02}}
|date=2016-09-02}}
</ref><ref>{{cite news |date=2016-08-31 |title=Life imitates satire: Facebook touts zlib killer just like Silicon Valley's Pied Piper |url=https://www.theregister.co.uk/2016/08/31/facebook_open_source_database/ |access-date=2016-09-06 |publisher=[[The Register]]}}
</ref><ref>{{cite news
|url=https://www.theregister.co.uk/2016/08/31/facebook_open_source_database/
|date=2016-08-31
|publisher=The Register
⚫
|access-date=2016-09-06
|title=Life imitates satire: Facebook touts zlib killer just like Silicon Valley's Pied Piper}}
</ref>
</ref>


==Features==
==Features==
Zstandard was designed to give a [[Data compression ratio|compression ratio]] comparable to that of the [[DEFLATE]] algorithm (developed in 1991 and used in the original [[zip (file format)|ZIP]] and [[gzip]] programs), but faster, especially for decompression. It is tunable with compression levels ranging from negative 7 (fastest)<ref>https://github.com/facebook/zstd/releases/tag/v1.3.4 Faster compression levels</ref> to 22 (slowest in compression speed, but best compression ratio).
Zstandard was designed to give a [[Data compression ratio|compression ratio]] comparable to that of the [[DEFLATE]] algorithm (developed in 1991 and used in the original [[zip (file format)|ZIP]] and [[gzip]] programs), but faster, especially for decompression. It is tunable with compression levels ranging from negative 7 (fastest)<ref>{{Cite web |title=Release Zstandard v1.3.4 - faster everything · facebook/zstd |url=https://github.com/facebook/zstd/releases/tag/v1.3.4 |access-date=2024-03-27 |website=GitHub |language=en}}</ref> to 22 (slowest in compression speed, but best compression ratio).


The zstd package includes parallel (multi-threaded) implementations of both compression and decompression{{Citation needed|reason=No zstd implementation reviewed featured multithreaded decompression|date=April 2021}}. Starting from version 1.3.2 (October 2017), zstd optionally implements very long range search and deduplication ({{code|--long}}, 128&nbsp;MiB window) similar to [[rzip]] or [[lrzip]].<ref>{{cite web |title=Command Line Interface for Zstandard library |url=https://github.com/facebook/zstd/blob/dev/programs/README.md |website=GitHub |date=28 October 2021 |language=en}}</ref>
Starting from version 1.3.2 (October 2017), zstd optionally implements very long range search and deduplication ({{code|--long}}, 128&nbsp;MiB window) similar to [[rzip]] or [[lrzip]].<ref>{{cite web |title=Command Line Interface for Zstandard library |url=https://github.com/facebook/zstd/blob/dev/programs/README.md |website=GitHub |date=28 October 2021 |language=en}}</ref>


Compression speed can vary by a factor of 20 or more between the fastest and slowest levels, while decompression is uniformly fast, varying by less than 20% between the fastest and slowest levels.<ref name=openzfs>
Compression speed can vary by a factor of 20 or more between the fastest and slowest levels, while decompression is uniformly fast, varying by less than 20% between the fastest and slowest levels.<ref name=openzfs>
Line 53: Line 47:
|access-date=2019-04-20
|access-date=2019-04-20
|website=open-zfs.org
|website=open-zfs.org
}}</ref> Zstandard command-line has an "adaptive" ({{code|--adapt}}) mode that varies compression level depending on I/O conditions, mainly how fast it can write the output.
}}</ref> The Zstandard command-line has an "adaptive" ({{code|--adapt}}) mode that varies compression level depending on I/O conditions, mainly how fast it can write the output.


''Zstd'' at its maximum compression level gives a compression ratio close to [[Lempel–Ziv–Markov chain algorithm|lzma]],
''Zstd'' at its maximum compression level gives a compression ratio close to [[Lempel–Ziv–Markov chain algorithm|lzma]],
[[Lempel–Ziv–Markov chain algorithm#LZHAM|lzham]], and [[Prediction by partial matching#PPMX|ppmx]], and performs better than [[LHA (file format)|lza]], or [[bzip2]].<ref>
[[LZHAM|lzham]], and [[Prediction by partial matching#PPMX|ppmx]], and {{synthesis inline span|performs better{{vague|date=January 2023}} than [[LHA (file format)|lza]] or [[bzip2]].|date=January 2023}}<ref>
{{cite web
{{cite web
|url=http://mattmahoney.net/dc/silesia.html
|url=http://mattmahoney.net/dc/silesia.html
Line 68: Line 62:
|date=2016-08-29
|date=2016-08-29
|access-date=2016-09-01}}
|access-date=2016-09-01}}
</ref> Zstandard reaches the current [[Pareto efficiency|Pareto frontier]], as it decompresses faster than any other currently-available algorithm with similar or better compression ratio.<ref name="web-compression">
</ref> Zstandard reaches the current [[Pareto efficiency|Pareto frontier]], as it decompresses faster than any other currently available algorithm with similar or better compression ratio.<ref name="web-compression">
{{citation
{{citation
|title=TurboBench: Static/Dynamic web content compression benchmark
|title=TurboBench: Static/Dynamic web content compression benchmark
Line 80: Line 74:
</ref>
</ref>


Dictionaries can have a large impact on the compression ratio of small files, so Zstandard can use a user-provided compression dictionary. It also offers a training mode, able to generate a dictionary from a set of samples.<ref>
Dictionaries can have a large impact on the compression ratio of small files, so Zstandard can use a user-provided compression dictionary. It also offers a training mode, able to generate a dictionary from a set of samples.<ref>{{Cite web |date=2017-10-11 |title=Facebook developers report massive speedups and compression ratio improvements when using dictionaries |url=https://indico.fnal.gov/event/15154/contribution/5/material/slides/0.pdf |access-date=2024-03-27 |website=[[Fermilab]] |format=PDF}}</ref><ref>
⚫
{{cite web
|url=https://indico.fnal.gov/event/15154/contribution/5/material/slides/0.pdf
|title=Facebook developers report massive speedups and compression ratio improvements when using dictionaries}}
</ref><ref>
{{cite web
{{cite web
|url=https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/
|url=https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/
Line 94: Line 84:
==Design==
==Design==


Zstandard combines a dictionary-matching stage ([[LZ77]]) with a large search window and a fast [[entropy encoding|entropy coding]] stage, using both [[Finite State Entropy]] (a fast tabled version of ANS, [[asymmetric numeral systems#Tabled variant (tANS)|tANS]], used for entries in the Sequences section), and [[Huffman coding]] (used for entries in the Literals section).<ref>{{cite web|url=https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md#entropy-encoding|title=facebook/zstd|website=GitHub|date=28 October 2021}}</ref>
Zstandard combines a dictionary-matching stage ([[LZ77]]) with a large search window and a fast [[entropy encoding|entropy-coding]] stage. It uses both [[Huffman coding]] (used for entries in the Literals section)<ref>{{cite web|url=https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md#entropy-encoding|title=facebook/zstd|website=GitHub|date=28 October 2021}}</ref> and finite-state entropy (FSE) - a fast tabled version of ANS, [[asymmetric numeral systems#Tabled variant (tANS)|tANS]], used for entries in the Sequences section. Because of the way that FSE carries over state between symbols, decompression involves processing symbols within the Sequences section of each block in reverse order (from last to first).

Because of the way that FSE carries over state between symbols, decompression involves processing symbols within the Sequences section of each block in reverse order (from last to first).


==Usage==
==Usage==
Line 102: Line 90:
{{Infobox file format
{{Infobox file format
| name = Zstandard
| name = Zstandard
| extension = .zst<ref name="RFC8478">
| extension = .zst<ref name="RFC8878">
{{cite IETF
{{cite IETF
|title=Zstandard Compression and the application/zstd Media Type
|title=Zstandard Compression and the application/zstd Media Type
|rfc=8478
|rfc=8878
|last1=Collet
|last1=Collet
|first1=Yann
|first1=Yann
|editor1-last=Kucherawy
|editor1-last=Kucherawy
|editor1-first=Murray S.
|editor1-first=Murray S.
|editor1-link=Murray Kucherawy
⚫
|date=October 2018
|access-date=7 October 2020
|date=February 2021
|access-date=26 February 2023
|series=Internet Engineering Task Force Request for Comments}}
|series=Internet Engineering Task Force Request for Comments}}
</ref>
</ref>
| mime = application/zstd<ref name="RFC8478"/>
| mime = application/zstd<ref name="RFC8878"/>
| genre = [[Data compression]]
| genre = [[Data compression]]
| magic = <code>28 b5 2f fd</code><ref name="RFC8478"/>
| magic = <code>28 b5 2f fd</code><ref name="RFC8878"/>
| standard = {{IETF RFC|8478}}
| standard = {{IETF RFC|8878}}
| url = https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md
| url = {{URL|https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md}}
}}
}}


{{Infobox file format
{{Infobox file format
| name = Zstandard Dictionary
| name = Zstandard Dictionary
| magic = <code>37 a4 30 ec</code><ref name="RFC8478"/>
| magic = <code>37 a4 30 ec</code><ref name="RFC8878"/>
| standard = {{IETF RFC|8478}}
| standard = {{IETF RFC|8878}}
| url = https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#dictionary-format
| url = {{URL|https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#dictionary-format}}
}}
}}


The [[Linux kernel]] has included Zstandard since November 2017 (version 4.14) as a compression method for the [[btrfs]] and [[squashfs]] filesystems.<ref>{{cite web
The [[Linux kernel]] has included Zstandard since November 2017 (version 4.14) as a compression method for the [[btrfs]] and [[squashfs]] filesystems.<ref>{{cite web |last=Corbet |first=Jonathan |date=2017-09-17 |title=The rest of the 4.14 merge window [LWN.net] |url=https://lwn.net/Articles/733846/ |access-date=2024-03-27 |website=[[lwn.net]]}}
|url=https://lwn.net/Articles/733846/
|title=The rest of the 4.14 merge window [LWN.net]
|website=lwn.net}}
</ref><ref>
</ref><ref>
{{cite web |date=2017-12-30 |title=Linux_4.14 - Linux Kernel Newbies |url=https://kernelnewbies.org/Linux_4.14 |access-date=2018-08-16 |publisher=Kernelnewbies.org}}
{{cite web
|url=https://kernelnewbies.org/Linux_4.14
|title=Linux_4.14 - Linux Kernel Newbies
|publisher=Kernelnewbies.org
⚫
|access-date=2018-08-16
}}
</ref><ref>
</ref><ref>
{{cite web |last=Larabel |first=Michael |date=2017-09-08 |title=Zstd Compression For Btrfs & Squashfs Set For Linux 4.14, Already Used Within Facebook - Phoronix |url=https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.14-Zstd-Pull |website=www.phoronix.com}}</ref>
{{cite web
|url=https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.14-Zstd-Pull
|title=Zstd Compression For Btrfs & Squashfs Set For Linux 4.14, Already Used Within Facebook - Phoronix
|website=www.phoronix.com
}}</ref>


In 2017, Allan Jude integrated Zstandard into the FreeBSD kernel<ref>{{cite web | url=https://github.com/freebsd/freebsd/commit/28ef16535cde21eeeaf75f6006b3a77952b3b51 | title=Integrate ZSTD into the kernel · freebsd/Freebsd-SRC@28ef165 | website=[[GitHub]] }}</ref> and it was subsequently integrated as a compressor option for core dumps (both user programs and kernel panics). It was also used to create a proof of concept [[OpenZFS]] compression method<ref name=openzfs /> which was integrated in 2020.<ref>{{Cite web|url=https://github.com/openzfs/zfs/commit/10b3c7f5e424f54b3ba82dbf1600d866e64ec0a0|title=Add ZSTD support to ZFS · openzfs/ZFS@10b3c7f|website=[[GitHub]]}}</ref>
In 2017, Allan Jude integrated Zstandard into the [[FreeBSD]] kernel,<ref>{{cite web | url=https://github.com/freebsd/freebsd/commit/28ef16535cde21eeeaf75f6006b3a77952b3b51 | title=Integrate ZSTD into the kernel · freebsd/Freebsd-SRC@28ef165 | website=[[GitHub]] }}</ref> and it was subsequently integrated as a compressor option for core dumps (both user programs and kernel panics). It was also used to create a proof-of-concept [[OpenZFS]] compression method<ref name=openzfs /> which was integrated in 2020.<ref>{{cite web|url=https://github.com/openzfs/zfs/commit/10b3c7f5e424f54b3ba82dbf1600d866e64ec0a0|title=Add ZSTD support to ZFS · openzfs/ZFS@10b3c7f|website=[[GitHub]]}}</ref>


The [[AWS Redshift]] and [[RocksDB]] databases include support for field compression using Zstandard.<ref name=Redshift>
The [[AWS Redshift]] and [[RocksDB]] databases include support for field compression using Zstandard.<ref name=Redshift>
Line 155: Line 132:
}}<!-- root? hadoop? MyRocks ? SQL servers? --></ref>
}}<!-- root? hadoop? MyRocks ? SQL servers? --></ref>


In March 2018, [[Canonical (company)|Canonical]] tested<ref>{{Cite web |url=https://www.phoronix.com/scan.php?page=news_item&px=Ubuntu-Zstd-Deb-Packages |title=Canonical Working On Zstd-Compressed Debian Packages For Ubuntu |last=Larabel |first=Michael |date=12 March 2018 |website=phoronix.com |publisher=Phoronix Media |access-date=29 October 2019 |quote=The developers at Canonical are considering a feature freeze exception to get this newly-developed Zstd Apt/Dpkg support in Ubuntu 18.04 LTS. In doing so, they mention they would be looking at enabling Zstd compression for packages by default in Ubuntu 18.10.}}</ref> the use of zstd as a [[deb (file format)|deb package]] compression method by default for the [[Ubuntu (operating system)|Ubuntu]] Linux distribution. Compared with [[XZ Utils|xz]] compression of deb packages, zstd at level 19 decompresses significantly faster, but at the cost of 6% larger package files. Debian developer [[Ian Jackson]] favored waiting several years before official adoption.<ref>{{cite web
In March 2018, [[Canonical (company)|Canonical]] tested<ref name="ubuntudeb">{{cite web |url=https://www.phoronix.com/scan.php?page=news_item&px=Ubuntu-Zstd-Deb-Packages |title=Canonical Working On Zstd-Compressed Debian Packages For Ubuntu |last=Larabel |first=Michael |date=12 March 2018 |website=phoronix.com |publisher=Phoronix Media |access-date=29 October 2019 |quote=The developers at Canonical are considering a feature freeze exception to get this newly-developed Zstd Apt/Dpkg support in Ubuntu 18.04 LTS. In doing so, they mention they would be looking at enabling Zstd compression for packages by default in Ubuntu 18.10.}}</ref> the use of zstd as a [[deb (file format)|deb package]] compression method by default for the [[Ubuntu (operating system)|Ubuntu]] Linux distribution. Compared with [[XZ Utils|xz]] compression of deb packages, zstd at level 19 decompresses significantly faster, but at the cost of 6% larger package files. Support was added to Debian (and subsequently, Ubuntu) in April 2018 (in version 1.6~rc1).<ref>{{cite web
|title=New Ubuntu Installs Could Be Speed Up by 10% with the Zstd Compression Algorithm
|title=New Ubuntu Installs Could Be Speed Up by 10% with the Zstd Compression Algorithm
|url=https://news.softpedia.com/news/new-ubuntu-installs-could-be-speed-up-by-10-with-the-zstd-compression-algorithm-520177.shtml
|url=https://news.softpedia.com/news/new-ubuntu-installs-could-be-speed-up-by-10-with-the-zstd-compression-algorithm-520177.shtml
Line 161: Line 138:
|access-date=13 August 2018
|access-date=13 August 2018
|date=Mar 12, 2018}}
|date=Mar 12, 2018}}
</ref><ref>
</ref><ref name="ubuntudeb"/><ref>
{{cite web
{{cite web
|title=Canonical Working On Zstd-Compressed Debian Packages For Ubuntu
|title=Debian Changelog for apt
|url=https://metadata.ftp-master.debian.org/changelogs//main/a/apt/apt_1.8.2.3_changelog
|url=https://www.phoronix.com/scan.php?page=news_item&px=Ubuntu-Zstd-Deb-Packages
|website=phoronix
|website=Debian
|access-date=13 August 2018
|access-date=7 November 2022
|language=en
|language=en
|date=12 March 2018
|date=19 April 2021
}}</ref><ref>
}}</ref>

{{cite mailing list|url=https://lists.debian.org/debian-devel/2018/04/msg00680.html |title=RFC: Support for zstd in .deb packages? |first=Ian |last=Jackson |author-link=Ian Jackson |date=2018-04-27 |mailing-list=debian-devel}}
⚫
In 2018 the algorithm was published as {{IETF RFC|8478}}, which also defines an associated [[media type]] "application/zstd", [[filename extension]] "zst", and [[HTTP compression|HTTP content encoding]] "zstd".<ref name="RFC8478">
⚫
{{cite IETF
|title=Zstandard Compression and the application/zstd Media Type
|rfc=8478
|last1=Collet
|first1=Yann
|editor1-last=Kucherawy
|editor1-first=Murray S.
|editor1-link=Murray Kucherawy
⚫
|date=October 2018
⚫
|access-date=7 October 2020
|series=Internet Engineering Task Force Request for Comments}}
</ref>
</ref>


⚫
[[Arch Linux]] added support for zstd as a package compression method in October 2019 with the release of the [[Pacman (software)|pacman]]&nbsp;5.2 package manager<ref>{{cite web |last=Larabel |first=Michael |date=2019-10-16 |title=Arch Linux Nears Roll-Out of ZSTD Compressed Packages for Faster Pacman Installs |url=https://www.phoronix.com/scan.php?page=news_item&px=Arch-Linux-Pacman-Zstd-Near |website=Phoronix}}</ref> and in January 2020 switched from xz to zstd for the packages in the official repository. Arch uses <code>zstd -c -T0 --ultra -20 -</code>, the size of all compressed packages combined increased by 0.8% (compared to xz), the decompression speed is 14<!-- 1300% --> times faster, decompression memory increased by 50&nbsp;MiB when using multiple threads, compression memory increases but scales with the number of threads used.<ref>
⚫
In 2018 the algorithm was published as {{IETF RFC|8478}}, which also defines an associated [[media type]] "application/zstd", [[filename extension]] "zst", and [[HTTP compression|HTTP content encoding]] "zstd".<ref name="RFC8478" />
⚫
{{cite web |url=https://www.archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ |title=Now using Zstandard instead of xz for package compression |last=Broda |first=Mara |date=2020-01-04 |website=Arch Linux |access-date=2020-01-05}}

⚫
[[Arch Linux]] added support for zstd as a package compression method in October 2019 with the release of the [[Arch Linux#Pacman|pacman]] 5.2 package manager,<ref>{{Cite web|url=https://www.phoronix.com/scan.php?page=news_item&px=Arch-Linux-Pacman-Zstd-Near|title = Arch Linux Nears Roll-Out of ZSTD Compressed Packages for Faster Pacman Installs - Phoronix}}</ref> and in January 2020 switched from xz to zstd for the packages in the official repository. Arch uses <code>zstd -c -T0 --ultra -20 -</code>, the size of all compressed packages combined increased by 0.8% (compared to xz), the decompression speed is 14 times faster, decompression memory increased by 50 MiB when using multiple threads, compression memory increases but scales with the number of threads used.<ref>
⚫
{{Cite web |url=https://www.archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ |title=Now using Zstandard instead of xz for package compression |last=Broda |first=Mara |date=2020-01-04 |website=Arch Linux |access-date=2020-01-05}}
</ref><ref>
</ref><ref>
{{cite mailing list|url=https://lists.archlinux.org/pipermail/arch-dev-public/2019-March/029542.html |mailing-list=arch-dev-public |title=RFC: (devtools) Changing default compression method to zstd |first=Mara |last=Broda |date=2019-03-25}}
{{cite mailing list|url=https://lists.archlinux.org/pipermail/arch-dev-public/2019-March/029542.html |mailing-list=arch-dev-public |title=RFC: (devtools) Changing default compression method to zstd |first=Mara |last=Broda |date=2019-03-25}}
</ref><ref>
</ref><ref>
{{cite web|url=https://git.archlinux.org/devtools.git/commit/?id=bcda211dd86b3bf54a9bc40d2e19f1aad4bbfbb8 |title=makepkg.conf: change default compression method to zstd |first1=Mara |last1=Broda |first2=Levente |last2=Polyak |date=2019-12-27}}
{{cite web |last1=Broda |first1=Mara |last2=Polyak |first2=Levente |date=2019-12-27 |title=makepkg.conf: change default compression method to zstd |website=[[GitHub]] |url=https://github.com/archlinux/devtools/commit/bcda211dd86b3bf54a9bc40d2e19f1aad4bbfbb8}}
</ref> Arch Linux later also switched to zstd as default compression algorithm for mkinitcpio [[initial ramdisk]] generator.<ref>{{Cite web|last=Razzolini|first=Giancarlo|date=2021-02-19|title=Arch Linux - News: Moving to Zstandard images by default on mkinitcpio|url=https://archlinux.org/news/moving-to-zstandard-images-by-default-on-mkinitcpio/|url-status=live|access-date=2021-12-28|website=Arch Linux}}</ref>
</ref> Arch Linux later also switched to zstd as default compression algorithm for mkinitcpio [[initial ramdisk]] generator.<ref>{{cite web|last=Razzolini|first=Giancarlo|date=2021-02-19|title=News: Moving to Zstandard images by default on mkinitcpio|url=https://archlinux.org/news/moving-to-zstandard-images-by-default-on-mkinitcpio/|access-date=2021-12-28|website=Arch Linux}}</ref>


[[Fedora (operating system)|Fedora]] added ZStandard support to [[RPM Package Manager|RPM]] in May 2018 (Fedora release 28), and used it for packaging the release in October 2019 (Fedora 31).<ref>{{Cite web|url=https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression|title = Changes/Switch RPMS to ZSTD compression - Fedora Project Wiki}}</ref> In Fedora 33, the filesystem is compressed by default with zstd.<ref>{{Cite web |date=2021-04-14 |title=Fedora Workstation 34 feature focus: Btrfs transparent compression |url=https://fedoramagazine.org/fedora-workstation-34-feature-focus-btrfs-transparent-compression/ |access-date=2022-05-12 |website=Fedora Magazine |language=en-US}}</ref><ref>{{Cite web |title=Changes/BtrfsTransparentCompression - Fedora Project Wiki |url=https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression |access-date=2022-05-12 |website=fedoraproject.org}}</ref>
[[Fedora (operating system)|Fedora]] added ZStandard support to [[RPM Package Manager|RPM]] in May 2018 (Fedora release 28) and used it for packaging the release in October 2019 (Fedora&nbsp;31).<ref>{{cite web |url=https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression |title = Changes/Switch RPMS to ZSTD compression |website=Fedora Project Wiki}}</ref> In Fedora&nbsp;33, the filesystem is compressed by default with zstd.<ref>{{cite web |date=2021-04-14 |title=Fedora Workstation 34 feature focus: Btrfs transparent compression |url=https://fedoramagazine.org/fedora-workstation-34-feature-focus-btrfs-transparent-compression/ |access-date=2022-05-12 |website=Fedora Magazine |language=en-US}}</ref><ref>{{cite web |title=Changes/BtrfsTransparentCompression |website=Fedora Project Wiki |url=https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression |access-date=2022-05-12}}</ref>


Full implementation of the algorithm with an option to choose the compression level is used in the .NSZ / .XCZ<ref>{{Cite web |url=https://gbatemp.net/threads/nszip-nsp-compressor-decompressor-to-reduce-storage.530313/ |title=RELEASE - nsZip - NSP compressor/decompressor to reduce storage |website=GBAtemp.net - The Independent Video Game Community |language=en-US |access-date=2019-11-03}}</ref> file formats, developed by the [[homebrew (video games)|homebrew]] community for the [[Nintendo Switch]] hybrid game console.<ref>{{Citation |last=Bosshard |first=Nico |title=nsZip is a tool to compress/decompress Nintendo Switch games using the here specified NSZ file format: nicoboss/nsZip |date=2019-10-31 |url=https://github.com/nicoboss/nsZip |access-date=2019-11-03}}</ref>
Full implementation of the algorithm with an option to choose the compression level is used in the .NSZ/.XCZ<ref>{{cite web |url=https://gbatemp.net/threads/nszip-nsp-compressor-decompressor-to-reduce-storage.530313/ |title=RELEASE - nsZip - NSP compressor/decompressor to reduce storage |website=GBAtemp.net - The Independent Video Game Community |date=20 October 2019 |language=en-US |access-date=2019-11-03}}</ref> file formats developed by the [[homebrew (video games)|homebrew]] community for the [[Nintendo Switch]] hybrid game console.<ref>{{Citation |last=Bosshard |first=Nico |title=nsZip is a tool to compress/decompress Nintendo Switch games using the here specified NSZ file format: nicoboss/nsZip |date=2019-10-31 |url=https://github.com/nicoboss/nsZip |access-date=2019-11-03}}</ref> Similarly, it is also one of many supported compression algorithms in the .RVZ [[Wii]] and [[GameCube]] [[Disk_image|disc image]] file format.


On 15 June 2020, Zstandard was implemented in version 6.3.8 of the zip file format with codec number&nbsp;93, deprecating the previous codec number of&nbsp;20 as it was implemented in version 6.3.7, released on 1 June.<ref name="zip638">{{citation |url=http://www.pkware.com/documents/APPNOTE/APPNOTE-6.3.8.TXT |title=APPNOTE.TXT - .ZIP File Format Specification Version: 6.3.8 |date=15 June 2020 |access-date=7 July 2020 }}</ref><ref name="zip637">{{citation |url=http://www.pkware.com/documents/APPNOTE/APPNOTE-6.3.7.TXT |title=APPNOTE.TXT - .ZIP File Format Specification Version: 6.3.7 |date=1 June 2020 |access-date=6 June 2020 }}</ref>
7-Zip ZS, a fork of [[7-Zip|7-Zip FM]] with Zstandard (and other formats) support, is developed by Tino Reichardt.<ref>{{Cite web|url=https://mcmilk.de/projects/7-Zip-zstd|title = Milkys Homepage - 7-Zip with support for Zstandard, Brotli, Lz4, Lz5 and Lizard Compression}}</ref>


In March 2024, [[Google Chrome]] version 123 (and [[Chromium (web browser)|Chromium]]-based browsers such as [[Brave (web browser)|Brave]] or [[Microsoft Edge]]) added zstd support in the [[HTTP header]] <code>Content-Encoding</code>.<ref>{{Cite web |date=2024-03-19 |orig-date= |title=New in Chrome 123 {{!}} Chrome Blog |url=https://developer.chrome.com/blog/new-in-chrome-123 |access-date=2024-04-16 |website=Chrome for Developers |language=en}}</ref> In May 2024, [[Firefox]] release 126.0 added zstd support in the [[HTTP header]] <code>Content-Encoding</code>.<ref>{{cite web |title=Firefox 126.0, See All New Features, Updates and Fixes |url=https://www.mozilla.org/en-US/firefox/126.0/releasenotes/ |access-date=15 May 2024}}</ref>
Modern7z, a Zstandard (and other formats) plugin for [[7-Zip|7-Zip FM]] is developed by Denis Anisimov (TC4shell).<ref>{{Cite web|url=https://www.tc4shell.com/en/7zip/modern7z|title = Modern7z}}</ref>


==License==
==License==
The reference implementation is licensed under the [[BSD licenses|BSD license]], published at [[GitHub]].<ref>{{cite news|url=http://www.zdnet.com/article/facebook-open-sources-zstandard-data-compression-algorithm-aims-to-replace-technology-behind-zip/|title=Facebook open sources Zstandard data compression algorithm, aims to replace technology behind Zip|date=August 31, 2016|publisher=ZDnet|access-date=2016-09-01}}</ref> Since version 1.0, it had an additional Grant of Patent Rights.<ref>[https://github.com/facebook/zstd/blob/v1.3.0/PATENTS zstd/PATENTS "Additional Grant of Patent Rights Version 2"], Facebook</ref>
The reference implementation is licensed under the [[BSD licenses|BSD license]], published at [[GitHub]].<ref>{{cite news|url=http://www.zdnet.com/article/facebook-open-sources-zstandard-data-compression-algorithm-aims-to-replace-technology-behind-zip/|title=Facebook open sources Zstandard data compression algorithm, aims to replace technology behind Zip|date=August 31, 2016|publisher=ZDnet|access-date=2016-09-01}}</ref> Since version 1.0, it had an additional Grant of Patent Rights.<ref>{{Cite web |title=zstd/PATENTS at v1.3.0 · facebook/zstd |url=https://github.com/facebook/zstd/blob/v1.3.0/PATENTS |access-date=2024-03-27 |website=GitHub |language=en}}</ref>


From version 1.3.1,<ref>[https://github.com/facebook/zstd/releases/tag/v1.3.1 "Zstd v1.3.1 release"], GitHub "facebook/zstd"</ref> this patent grant was dropped and the license was changed to a BSD + GPLv2 dual license.<ref>[https://github.com/facebook/zstd/pull/801 "New license"], GitHub "facebook/zstd"</ref>
From version 1.3.1,<ref>{{Cite web |title=Release Zstandard v1.3.1 · facebook/zstd |url=https://github.com/facebook/zstd/releases/tag/v1.3.1 |access-date=2024-03-27 |website=GitHub |language=en}}</ref> this patent grant was dropped and the license was changed to a BSD + GPLv2 dual license.<ref>{{Cite web |title=New license by Cyan4973 · Pull Request #801 · facebook/zstd |url=https://github.com/facebook/zstd/pull/801 |access-date=2024-03-27 |website=GitHub |language=en}}</ref>


==See also==
==See also==
⚫
* [[Zlib]]
⚫
* [[LZFSE]] – a similar algorithm by Apple used since iOS 9 and OS X 10.11 made open source on 1 June 2016
* [[LZ4 (compression algorithm)]] – a fast member of the LZ77 family
* [[LZ4 (compression algorithm)]] – a fast member of the LZ77 family
⚫
* [[LZFSE]] – a similar algorithm by Apple used since iOS 9 and OS X 10.11 and made open source on 1 June 2016
⚫
* [[Zlib]]


==References==
==References==
{{Reflist}}
{{Reflist|colwidth=30em}}


==External links==
==External links==
* {{official website}}
* {{Official website}}
* {{github|facebook/zstd}}
* {{Github|facebook/zstd}}
* {{GitHub|mcmilk/7-Zip-zstd|7zip with Zstandard}}
* {{GitHub|mcmilk/7-Zip-zstd|7zip with Zstandard}}
* "[https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/ Smaller and faster data compression with Zstandard]", Yann Collet and Chip Turner, 31 August 2016, Facebook Announcement
* "[https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/ Smaller and faster data compression with Zstandard]", Yann Collet and Chip Turner, 31 August 2016, Facebook Announcement
* [https://www.theguardian.com/info/developer-blog/2016/dec/01/discover-new-compression-iinovations-brotli-and-zstandard The Guardian is using ZStandard instead of zlib]
* [https://www.theguardian.com/info/developer-blog/2016/dec/01/discover-new-compression-iinovations-brotli-and-zstandard The Guardian is using ZStandard instead of zlib]


{{Compression Methods}}
{{Compression methods}}


⚫
[[Category:Lossless compression algorithms]]
⚫
[[Category:Free data compression software]]
⚫
[[Category:C (programming language) libraries]]
[[Category:2016 software]]
[[Category:2016 software]]
⚫
[[Category:C (programming language) libraries]]
⚫
[[Category:Free data compression software]]
⚫
[[Category:Lossless compression algorithms]]
[[Category:Software using the BSD license]]
[[Category:Software using the BSD license]]

Latest revision as of 15:39, 14 June 2024

".zst" redirects here. For the game console emulator also using this file extension, see ZSNES.

Original author(s)Yann Collet
Developer(s)Yann Collet, Nick Terrell, Przemysław Skibiński[1]
Initial release23 January 2015 (2015-01-23)
Stable release
1.5.6[2] Edit this on Wikidata / 27 March 2024; 2 months ago (27 March 2024)
Repository
Written inC
Operating systemCross-platform
PlatformPortable
TypeData compression
LicenseBSD-3-Clause or GPL-2.0-or-later (dual-licensed)
Websitefacebook.github.io/zstd/ Edit this on Wikidata

Zstandard is a lossless data compression algorithm developed by Yann Collet at Facebook. Zstd is the corresponding reference implementation in C, released as open-source software on 31 August 2016.[3][4]

Features[edit]

Zstandard was designed to give a compression ratio comparable to that of the DEFLATE algorithm (developed in 1991 and used in the original ZIP and gzip programs), but faster, especially for decompression. It is tunable with compression levels ranging from negative 7 (fastest)[5] to 22 (slowest in compression speed, but best compression ratio).

Starting from version 1.3.2 (October 2017), zstd optionally implements very long range search and deduplication (--long, 128 MiB window) similar to rzip or lrzip.[6]

Compression speed can vary by a factor of 20 or more between the fastest and slowest levels, while decompression is uniformly fast, varying by less than 20% between the fastest and slowest levels.[7] The Zstandard command-line has an "adaptive" (--adapt) mode that varies compression level depending on I/O conditions, mainly how fast it can write the output.

Zstd at its maximum compression level gives a compression ratio close to lzma, lzham, and ppmx, and performs better[vague] than lza or bzip2.[improper synthesis?][8][9] Zstandard reaches the current Pareto frontier, as it decompresses faster than any other currently available algorithm with similar or better compression ratio.[10][11]

Dictionaries can have a large impact on the compression ratio of small files, so Zstandard can use a user-provided compression dictionary. It also offers a training mode, able to generate a dictionary from a set of samples.[12][13] In particular, one dictionary can be loaded to process large sets of files with redundancy between files, but not necessarily within each file, e.g., log files.

Design[edit]

Zstandard combines a dictionary-matching stage (LZ77) with a large search window and a fast entropy-coding stage. It uses both Huffman coding (used for entries in the Literals section)[14] and finite-state entropy (FSE) - a fast tabled version of ANS, tANS, used for entries in the Sequences section. Because of the way that FSE carries over state between symbols, decompression involves processing symbols within the Sequences section of each block in reverse order (from last to first).

Usage[edit]

Zstandard
Filename extension
.zst[15]
Internet media type
application/zstd[15]
Magic number28 b5 2f fd[15]
Type of formatData compression
StandardRFC 8878
Websitegithub.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md
Zstandard Dictionary
Internet media typeapplication/zstd
Magic number37 a4 30 ec[15]
StandardRFC 8878
Websitegithub.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#dictionary-format

The Linux kernel has included Zstandard since November 2017 (version 4.14) as a compression method for the btrfs and squashfs filesystems.[16][17][18]

In 2017, Allan Jude integrated Zstandard into the FreeBSD kernel,[19] and it was subsequently integrated as a compressor option for core dumps (both user programs and kernel panics). It was also used to create a proof-of-concept OpenZFS compression method[7] which was integrated in 2020.[20]

The AWS Redshift and RocksDB databases include support for field compression using Zstandard.[21]

In March 2018, Canonical tested[22] the use of zstd as a deb package compression method by default for the Ubuntu Linux distribution. Compared with xz compression of deb packages, zstd at level 19 decompresses significantly faster, but at the cost of 6% larger package files. Support was added to Debian (and subsequently, Ubuntu) in April 2018 (in version 1.6~rc1).[23][22][24]

In 2018 the algorithm was published as RFC 8478, which also defines an associated media type "application/zstd", filename extension "zst", and HTTP content encoding "zstd".[25]

Arch Linux added support for zstd as a package compression method in October 2019 with the release of the pacman 5.2 package manager[26] and in January 2020 switched from xz to zstd for the packages in the official repository. Arch uses zstd -c -T0 --ultra -20 -, the size of all compressed packages combined increased by 0.8% (compared to xz), the decompression speed is 14 times faster, decompression memory increased by 50 MiB when using multiple threads, compression memory increases but scales with the number of threads used.[27][28][29] Arch Linux later also switched to zstd as default compression algorithm for mkinitcpio initial ramdisk generator.[30]

Fedora added ZStandard support to RPM in May 2018 (Fedora release 28) and used it for packaging the release in October 2019 (Fedora 31).[31] In Fedora 33, the filesystem is compressed by default with zstd.[32][33]

Full implementation of the algorithm with an option to choose the compression level is used in the .NSZ/.XCZ[34] file formats developed by the homebrew community for the Nintendo Switch hybrid game console.[35] Similarly, it is also one of many supported compression algorithms in the .RVZ Wii and GameCube disc image file format.

On 15 June 2020, Zstandard was implemented in version 6.3.8 of the zip file format with codec number 93, deprecating the previous codec number of 20 as it was implemented in version 6.3.7, released on 1 June.[36][37]

In March 2024, Google Chrome version 123 (and Chromium-based browsers such as Brave or Microsoft Edge) added zstd support in the HTTP header Content-Encoding.[38] In May 2024, Firefox release 126.0 added zstd support in the HTTP header Content-Encoding.[39]

License[edit]

The reference implementation is licensed under the BSD license, published at GitHub.[40] Since version 1.0, it had an additional Grant of Patent Rights.[41]

From version 1.3.1,[42] this patent grant was dropped and the license was changed to a BSD + GPLv2 dual license.[43]

See also[edit]

References[edit]

  1. ^ "Contributors to facebook/zstd". github.com. Archived from the original on 27 January 2021. Retrieved 26 January 2021.
  2. ^ "Release Zstandard v1.5.6 - Chrome Edition · facebook/zstd". Retrieved 27 March 2024.
  3. ^ Sergio De Simone (2 September 2016). "Facebook Open-Sources New Compression Algorithm Outperforming Zlib". InfoQ. Retrieved 20 April 2019.
  4. ^ "Life imitates satire: Facebook touts zlib killer just like Silicon Valley's Pied Piper". The Register. 31 August 2016. Retrieved 6 September 2016.
  5. ^ "Release Zstandard v1.3.4 - faster everything · facebook/zstd". GitHub. Retrieved 27 March 2024.
  6. ^ "Command Line Interface for Zstandard library". GitHub. 28 October 2021.
  7. ^ a b "ZStandard in ZFS" (PDF). open-zfs.org. 2017. Retrieved 20 April 2019.
  8. ^ Matt Mahoney. "Silesia Open Source Compression Benchmark". Retrieved 10 May 2019.
  9. ^ Matt Mahoney (29 August 2016). "Large Text Compression Benchmark, .2157 zstd". Retrieved 1 September 2016.
  10. ^ TurboBench: Static/Dynamic web content compression benchmark, PowTurbo
  11. ^ Matt Mahoney, Silesia Open Source Compression Benchmark
  12. ^ "Facebook developers report massive speedups and compression ratio improvements when using dictionaries" (PDF). Fermilab. 11 October 2017. Retrieved 27 March 2024.
  13. ^ "Smaller and faster data compression with Zstandard". Facebook. 31 August 2016.
  14. ^ "facebook/zstd". GitHub. 28 October 2021.
  15. ^ a b c d Collet, Yann (February 2021). Kucherawy, Murray S. (ed.). Zstandard Compression and the application/zstd Media Type. Internet Engineering Task Force Request for Comments. doi:10.17487/RFC8878. RFC 8878. Retrieved 26 February 2023.
  16. ^ Corbet, Jonathan (17 September 2017). "The rest of the 4.14 merge window [LWN.net]". lwn.net. Retrieved 27 March 2024.
  17. ^ "Linux_4.14 - Linux Kernel Newbies". Kernelnewbies.org. 30 December 2017. Retrieved 16 August 2018.
  18. ^ Larabel, Michael (8 September 2017). "Zstd Compression For Btrfs & Squashfs Set For Linux 4.14, Already Used Within Facebook - Phoronix". www.phoronix.com.
  19. ^ "Integrate ZSTD into the kernel · freebsd/Freebsd-SRC@28ef165". GitHub.
  20. ^ "Add ZSTD support to ZFS · openzfs/ZFS@10b3c7f". GitHub.
  21. ^ "Zstandard Encoding - Amazon Redshift". 20 April 2019.
  22. ^ a b Larabel, Michael (12 March 2018). "Canonical Working On Zstd-Compressed Debian Packages For Ubuntu". phoronix.com. Phoronix Media. Retrieved 29 October 2019. The developers at Canonical are considering a feature freeze exception to get this newly-developed Zstd Apt/Dpkg support in Ubuntu 18.04 LTS. In doing so, they mention they would be looking at enabling Zstd compression for packages by default in Ubuntu 18.10.
  23. ^ "New Ubuntu Installs Could Be Speed Up by 10% with the Zstd Compression Algorithm". Softpedia. 12 March 2018. Retrieved 13 August 2018.
  24. ^ "Debian Changelog for apt". Debian. 19 April 2021. Retrieved 7 November 2022.
  25. ^ Collet, Yann (October 2018). Kucherawy, Murray S. (ed.). Zstandard Compression and the application/zstd Media Type. Internet Engineering Task Force Request for Comments. doi:10.17487/RFC8478. RFC 8478. Retrieved 7 October 2020.
  26. ^ Larabel, Michael (16 October 2019). "Arch Linux Nears Roll-Out of ZSTD Compressed Packages for Faster Pacman Installs". Phoronix.
  27. ^ Broda, Mara (4 January 2020). "Now using Zstandard instead of xz for package compression". Arch Linux. Retrieved 5 January 2020.
  28. ^ Broda, Mara (25 March 2019). "RFC: (devtools) Changing default compression method to zstd". arch-dev-public (Mailing list).
  29. ^ Broda, Mara; Polyak, Levente (27 December 2019). "makepkg.conf: change default compression method to zstd". GitHub.
  30. ^ Razzolini, Giancarlo (19 February 2021). "News: Moving to Zstandard images by default on mkinitcpio". Arch Linux. Retrieved 28 December 2021.
  31. ^ "Changes/Switch RPMS to ZSTD compression". Fedora Project Wiki.
  32. ^ "Fedora Workstation 34 feature focus: Btrfs transparent compression". Fedora Magazine. 14 April 2021. Retrieved 12 May 2022.
  33. ^ "Changes/BtrfsTransparentCompression". Fedora Project Wiki. Retrieved 12 May 2022.
  34. ^ "RELEASE - nsZip - NSP compressor/decompressor to reduce storage". GBAtemp.net - The Independent Video Game Community. 20 October 2019. Retrieved 3 November 2019.
  35. ^ Bosshard, Nico (31 October 2019), nsZip is a tool to compress/decompress Nintendo Switch games using the here specified NSZ file format: nicoboss/nsZip, retrieved 3 November 2019
  36. ^ APPNOTE.TXT - .ZIP File Format Specification Version: 6.3.8, 15 June 2020, retrieved 7 July 2020
  37. ^ APPNOTE.TXT - .ZIP File Format Specification Version: 6.3.7, 1 June 2020, retrieved 6 June 2020
  38. ^ "New in Chrome 123 | Chrome Blog". Chrome for Developers. 19 March 2024. Retrieved 16 April 2024.
  39. ^ "Firefox 126.0, See All New Features, Updates and Fixes". Retrieved 15 May 2024.
  40. ^ "Facebook open sources Zstandard data compression algorithm, aims to replace technology behind Zip". ZDnet. 31 August 2016. Retrieved 1 September 2016.
  41. ^ "zstd/PATENTS at v1.3.0 · facebook/zstd". GitHub. Retrieved 27 March 2024.
  42. ^ "Release Zstandard v1.3.1 · facebook/zstd". GitHub. Retrieved 27 March 2024.
  43. ^ "New license by Cyan4973 · Pull Request #801 · facebook/zstd". GitHub. Retrieved 27 March 2024.

External links[edit]

Retrieved from "https://en.wikipedia.org/w/index.php?title=Zstd&oldid=1229050179"