Skip to content

Commit 38816f9

Browse files
Cyan4973Cyan4973
authored
Merge pull request #311 from Cyan4973/dev
Dev
2 parents 80b0e53 + 0588ee6 commit 38816f9

File tree

106 files changed

+2214
-4029
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

106 files changed

+2214
-4029
lines changed

‎.travis.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ matrix:
99
env: PLATFORM="Ubuntu 12.04 container" CMD="make test && make clean && make travis-install"
1010
- os: linux
1111
sudo: false
12-
env: PLATFORM="Ubuntu 12.04 container" CMD="make -C programs test-zstd_nolegacy && make clean && make zlibwrapper && make clean && make cmaketest"
12+
env: PLATFORM="Ubuntu 12.04 container" CMD="make -C tests test-zstd_nolegacy && make clean && make zlibwrapper && make clean && make cmaketest"
1313
- os: linux
1414
sudo: false
1515
env: PLATFORM="Ubuntu 12.04 container" CMD="make usan"
@@ -39,7 +39,7 @@ matrix:
3939
- qemu-user-static
4040
- os: linux
4141
sudo: required
42-
env: PLATFORM="Ubuntu 12.04" CMD="make -C programs test32"
42+
env: PLATFORM="Ubuntu 12.04" CMD="make -C tests test32"
4343
addons:
4444
apt:
4545
packages:
@@ -58,7 +58,7 @@ matrix:
5858
- gcc-multilib
5959
- os: linux
6060
sudo: required
61-
env: PLATFORM="Ubuntu 12.04" CMD="make -C programs valgrindTest"
61+
env: PLATFORM="Ubuntu 12.04" CMD="make -C tests valgrindTest"
6262
addons:
6363
apt:
6464
packages:

‎CONTRIBUTING.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Contributing to Zstandard
2+
We want to make contributing to this project as easy and transparent as
3+
possible.
4+
5+
## Our Development Process
6+
New versions are being developed in the "dev" branch,
7+
or in their own feature branch.
8+
When they are deemed ready for a release, they are merged into "master".
9+
10+
As a consequences, all contributions must stage first through "dev"
11+
or their own feature branch.
12+
13+
## Pull Requests
14+
We actively welcome your pull requests.
15+
16+
1. Fork the repo and create your branch from `dev`.
17+
2. If you've added code that should be tested, add tests.
18+
3. If you've changed APIs, update the documentation.
19+
4. Ensure the test suite passes.
20+
5. Make sure your code lints.
21+
6. If you haven't already, complete the Contributor License Agreement ("CLA").
22+
23+
## Contributor License Agreement ("CLA")
24+
In order to accept your pull request, we need you to submit a CLA. You only need
25+
to do this once to work on any of Facebook's open source projects.
26+
27+
Complete your CLA here: <https://code.facebook.com/cla>
28+
29+
## Issues
30+
We use GitHub issues to track public bugs. Please ensure your description is
31+
clear and has sufficient instructions to be able to reproduce the issue.
32+
33+
Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe
34+
disclosure of security bugs. In those cases, please go through the process
35+
outlined on that page and do not file a public issue.
36+
37+
## Coding Style
38+
* 4 spaces for indentation rather than tabs
39+
40+
## License
41+
By contributing to Zstandard, you agree that your contributions will be licensed
42+
under the [LICENSE](LICENSE) file in the root directory of this source tree.

‎lib/LICENSE‎ renamed to ‎LICENSE‎

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,22 @@
1-
ZSTD Library
2-
Copyright (c) 2014-2015, Yann Collet
3-
All rights reserved.
4-
51
BSD License
62

3+
For Zstandard software
4+
5+
Copyright (c) 2016-present, Facebook, Inc. All rights reserved.
6+
77
Redistribution and use in source and binary forms, with or without modification,
88
are permitted provided that the following conditions are met:
99

10-
* Redistributions of source code must retain the above copyright notice, this
11-
list of conditions and the following disclaimer.
10+
* Redistributions of source code must retain the above copyright notice, this
11+
list of conditions and the following disclaimer.
12+
13+
* Redistributions in binary form must reproduce the above copyright notice,
14+
this list of conditions and the following disclaimer in the documentation
15+
and/or other materials provided with the distribution.
1216

13-
* Redistributions in binary form must reproduce the above copyright notice, this
14-
list of conditions and the following disclaimer in the documentation and/or
15-
other materials provided with the distribution.
17+
* Neither the name Facebook nor the names of its contributors may be used to
18+
endorse or promote products derived from this software without specific
19+
prior written permission.
1620

1721
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
1822
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

‎LICENSE-examples‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
Copyright (c) 2016-present, Facebook, Inc. All rights reserved.
2+
3+
The examples provided by Facebook are for non-commercial testing and evaluation
4+
purposes only. Facebook reserves all rights not expressly granted.
5+
6+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
7+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
8+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
9+
FACEBOOK BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
10+
ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
11+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

‎Makefile‎

Lines changed: 20 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,16 @@
11
# ################################################################
2-
# zstd - Makefile
3-
# Copyright (C) Yann Collet 2014-2016
2+
# Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
43
# All rights reserved.
54
#
6-
# BSD license
7-
#
8-
# Redistribution and use in source and binary forms, with or without modification,
9-
# are permitted provided that the following conditions are met:
10-
#
11-
# * Redistributions of source code must retain the above copyright notice, this
12-
# list of conditions and the following disclaimer.
13-
#
14-
# * Redistributions in binary form must reproduce the above copyright notice, this
15-
# list of conditions and the following disclaimer in the documentation and/or
16-
# other materials provided with the distribution.
17-
#
18-
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
19-
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
20-
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
21-
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
22-
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
23-
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
24-
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
25-
# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
26-
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
27-
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
28-
#
29-
# You can contact the author at :
30-
# - zstd homepage : http://www.zstd.net/
5+
# This source code is licensed under the BSD-style license found in the
6+
# LICENSE file in the root directory of this source tree. An additional grant
7+
# of patent rights can be found in the PATENTS file in the same directory.
318
# ################################################################
329

3310
PRGDIR = programs
3411
ZSTDDIR = lib
3512
ZWRAPDIR = zlibWrapper
13+
TESTDIR = tests
3614

3715
# Define nul output
3816
ifneq (,$(filter Windows%,$(OS)))
@@ -47,9 +25,8 @@ default: zstd
4725

4826
all:
4927
$(MAKE) -C $(ZSTDDIR) $@
50-
$(MAKE) -C $(PRGDIR) $@
51-
@rm -f lib/decompress/*.o
52-
$(MAKE) -C $(PRGDIR) all32
28+
$(MAKE) -C $(PRGDIR) $@ zstd32
29+
$(MAKE) -C $(TESTDIR) $@ all32
5330

5431
zstd:
5532
$(MAKE) -C $(PRGDIR)
@@ -60,11 +37,12 @@ zlibwrapper:
6037
$(MAKE) -C $(ZWRAPDIR) all
6138

6239
test:
63-
$(MAKE) -C $(PRGDIR) $@
40+
$(MAKE) -C $(TESTDIR) $@
6441

6542
clean:
6643
@$(MAKE) -C $(ZSTDDIR) $@ > $(VOID)
6744
@$(MAKE) -C $(PRGDIR) $@ > $(VOID)
45+
@$(MAKE) -C $(TESTDIR) $@ > $(VOID)
6846
@$(MAKE) -C $(ZWRAPDIR) $@ > $(VOID)
6947
@rm -f zstd
7048
@echo Cleaning completed
@@ -99,19 +77,19 @@ gcc6test: clean
9977

10078
clangtest: clean
10179
clang -v
102-
$(MAKE) all CC=clang MOREFLAGS="-Werror -Wconversion -Wno-sign-conversion"
80+
$(MAKE) all CC=clang MOREFLAGS="-Werror -Wconversion -Wno-sign-conversion -Wdocumentation"
10381

10482
armtest: clean
105-
$(MAKE) -C $(PRGDIR) datagen # use native, faster
106-
$(MAKE) -C $(PRGDIR) test CC=arm-linux-gnueabi-gcc ZSTDRTTEST= MOREFLAGS="-Werror -static"
83+
$(MAKE) -C $(TESTDIR) datagen # use native, faster
84+
$(MAKE) -C $(TESTDIR) test CC=arm-linux-gnueabi-gcc ZSTDRTTEST= MOREFLAGS="-Werror -static"
10785

10886
ppctest: clean
109-
$(MAKE) -C $(PRGDIR) datagen # use native, faster
110-
$(MAKE) -C $(PRGDIR) test CC=powerpc-linux-gnu-gcc ZSTDRTTEST= MOREFLAGS="-Werror -Wno-attributes -static"
87+
$(MAKE) -C $(TESTDIR) datagen # use native, faster
88+
$(MAKE) -C $(TESTDIR) test CC=powerpc-linux-gnu-gcc ZSTDRTTEST= MOREFLAGS="-Werror -Wno-attributes -static"
11189

11290
ppc64test: clean
113-
$(MAKE) -C $(PRGDIR) datagen # use native, faster
114-
$(MAKE) -C $(PRGDIR) test CC=powerpc-linux-gnu-gcc ZSTDRTTEST= MOREFLAGS="-m64 -static"
91+
$(MAKE) -C $(TESTDIR) datagen # use native, faster
92+
$(MAKE) -C $(TESTDIR) test CC=powerpc-linux-gnu-gcc ZSTDRTTEST= MOREFLAGS="-m64 -static"
11593

11694
usan: clean
11795
$(MAKE) test CC=clang MOREFLAGS="-g -fsanitize=undefined"
@@ -123,7 +101,7 @@ msan: clean
123101
$(MAKE) test CC=clang MOREFLAGS="-g -fsanitize=memory -fno-omit-frame-pointer" # datagen.c fails this test for no obvious reason
124102

125103
asan32: clean
126-
$(MAKE) -C $(PRGDIR) test32 CC=clang MOREFLAGS="-g -fsanitize=address"
104+
$(MAKE) -C $(TESTDIR) test32 CC=clang MOREFLAGS="-g -fsanitize=address"
127105

128106
uasan: clean
129107
$(MAKE) test CC=clang MOREFLAGS="-g -fsanitize=address -fsanitize=undefined"
@@ -163,13 +141,13 @@ c11test: clean
163141
CFLAGS="-std=c11" $(MAKE) all
164142

165143
bmix64test: clean
166-
CFLAGS="-O3 -mbmi -Werror" $(MAKE) -C $(PRGDIR) test
144+
CFLAGS="-O3 -mbmi -Werror" $(MAKE) -C $(TESTDIR) test
167145

168146
bmix32test: clean
169-
CFLAGS="-O3 -mbmi -mx32 -Werror" $(MAKE) -C $(PRGDIR) test
147+
CFLAGS="-O3 -mbmi -mx32 -Werror" $(MAKE) -C $(TESTDIR) test
170148

171149
bmi32test: clean
172-
CFLAGS="-O3 -mbmi -m32 -Werror" $(MAKE) -C $(PRGDIR) test
150+
CFLAGS="-O3 -mbmi -m32 -Werror" $(MAKE) -C $(TESTDIR) test
173151

174152
staticAnalyze: clean
175153
CPPFLAGS=-g scan-build --status-bugs -v $(MAKE) all

‎NEWS‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,12 @@
1+
v1.0.0
2+
Change Licensing, all project is now BSD, Copyright Facebook
3+
Small decompression speed improvement
4+
API : Streaming API supports legacy format
5+
API : ZDICT_getDictID(), ZSTD_sizeof_{CCtx, DCtx, CStream, DStream}(), ZSTD_setDStreamParamter()
6+
CLI supports legacy formats v0.4+
7+
Fixed : compression fails on certain huge files, reported by Jesse McGrew
8+
Enhanced documentation, by Przemyslaw Skibinski
9+
110
v0.8.1
211
New streaming API
312
Changed : --ultra now enables levels beyond 19

‎PATENTS‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
Additional Grant of Patent Rights Version 2
2+
3+
"Software" means the Zstandard software distributed by Facebook, Inc.
4+
5+
Facebook, Inc. ("Facebook") hereby grants to each recipient of the Software
6+
("you") a perpetual, worldwide, royalty-free, non-exclusive, irrevocable
7+
(subject to the termination provision below) license under any Necessary
8+
Claims, to make, have made, use, sell, offer to sell, import, and otherwise
9+
transfer the Software. For avoidance of doubt, no license is granted under
10+
Facebook’s rights in any patent claims that are infringed by (i) modifications
11+
to the Software made by you or any third party or (ii) the Software in
12+
combination with any software or other technology.
13+
14+
The license granted hereunder will terminate, automatically and without notice,
15+
if you (or any of your subsidiaries, corporate affiliates or agents) initiate
16+
directly or indirectly, or take a direct financial interest in, any Patent
17+
Assertion: (i) against Facebook or any of its subsidiaries or corporate
18+
affiliates, (ii) against any party if such Patent Assertion arises in whole or
19+
in part from any software, technology, product or service of Facebook or any of
20+
its subsidiaries or corporate affiliates, or (iii) against any party relating
21+
to the Software. Notwithstanding the foregoing, if Facebook or any of its
22+
subsidiaries or corporate affiliates files a lawsuit alleging patent
23+
infringement against you in the first instance, and you respond by filing a
24+
patent infringement counterclaim in that lawsuit against that party that is
25+
unrelated to the Software, the license granted hereunder will not terminate
26+
under section (i) of this paragraph due to such counterclaim.
27+
28+
A "Necessary Claim" is a claim of a patent owned by Facebook that is
29+
necessarily infringed by the Software standing alone.
30+
31+
A "Patent Assertion" is any lawsuit or other action alleging direct, indirect,
32+
or contributory infringement or inducement to infringe any patent, including a
33+
cross-claim or counterclaim.

‎README.md‎

Lines changed: 12 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ As a reference, several fast compression algorithms were tested and compared on
1919
|Name | Ratio | C.speed | D.speed |
2020
|-----------------|-------|--------:|--------:|
2121
| | | MB/s | MB/s |
22-
|**zstd 0.8.0 -1**|**2.877**|**330**| **930** |
22+
|**zstd 0.8.2 -1**|**2.877**|**330**| **940** |
2323
| [zlib] 1.2.8 -1 | 2.730 | 95 | 360 |
2424
| brotli 0.4 -0 | 2.708 | 320 | 375 |
2525
| QuickLZ 1.5 | 2.237 | 510 | 605 |
@@ -61,8 +61,6 @@ Hence, deploying one dictionary per type of data will provide the greater benefi
6161

6262
#### Dictionary compression How To :
6363

64-
##### _Using the Command Line Utility_ :
65-
6664
1) Create the dictionary
6765

6866
`zstd --train FullPathToTrainingSet/* -o dictionaryName`
@@ -77,14 +75,19 @@ Hence, deploying one dictionary per type of data will provide the greater benefi
7775

7876
### Status
7977

80-
Zstd compression format has reached "Final status". It means it is planned to become the official stable zstd format tagged `v1.0`. The reason it's not yet tagged `v1.0` is that it currently performs its "validation period", making sure the format holds all its promises and nothing was missed.
81-
Zstd library also offers legacy decoder support. Any data compressed by any version >= `v0.1` is decodable now and in the future.
82-
The library has been validated using strong [fuzzer tests](https://en.wikipedia.org/wiki/Fuzz_testing), including both [internal tools](programs/fuzzer.c) and [external ones](http://lcamtuf.coredump.cx/afl). It's able to withstand hazard situations, including invalid inputs.
83-
As a consequence, Zstandard is considered safe for, and is currently used in, production environments.
78+
Zstandard is currently deployed within Facebook. It is used daily to compress and decompress very large amount of data in multiple formats and use cases.
79+
Zstandard is considered safe for production environments.
80+
81+
### License
82+
83+
Zstandard is [BSD-licensed](LICENSE). We also provide an [additional patent grant](PATENTS).
8484

85-
### Branch Policy
85+
### Contributing
8686

87-
The "dev" branch is the one where all contributions will be merged before reaching "master". If you plan to propose a patch, please commit into the "dev" branch or its own feature branch. Direct commit to "master" are not permitted.
87+
The "dev" branch is the one where all contributions will be merged before reaching "master".
88+
If you plan to propose a patch, please commit into the "dev" branch or its own feature branch.
89+
Direct commit to "master" are not permitted.
90+
For more information, please read [CONTRIBUTING](CONTRIBUTING.md).
8891

8992
### Miscellaneous
9093

0 commit comments

Comments
 (0)