Code

ZSTD_compressSequencesAndLiterals#4217

Code

Merged

Cyan4973 merged 70 commits intodevfacebook/zstd:devfrom ZSTD_compressSequencesAndLiteralsfacebook/zstd:ZSTD_compressSequencesAndLiteralsCopy head branch name to clipboardDec 26, 2024

+1,795-1,205

Commits

Commits on Dec 20, 2024

• publish new symbol ZSTD_compressSequencesAndLiterals()

  Cyan4973

  committed
  125f052

  Copy full SHA for 125f052

• created ZSTD_storeSeqOnly()

  Show description for a00f45a
  Cyan4973

  committed
  a00f45a

  Copy full SHA for a00f45a

• move Sequences definition to zstd_compress_internal.h

  Show description for b4a40a8
  Cyan4973

  committed
  b4a40a8

  Copy full SHA for b4a40a8

• codemod: seqDef -> SeqDef

  Show description for 9671813
  Cyan4973

  committed
  9671813

  Copy full SHA for 9671813

• codemod: seqStore_t -> SeqStore_t

  Show description for a224572
  Cyan4973

  committed
  a224572

  Copy full SHA for a224572

• codemod: ZSTD_sequenceLength -> ZSTD_SequenceLength

  Cyan4973

  committed
  8d4506b

  Copy full SHA for 8d4506b

• codemod: symbolEncodingType_e -> SymbolEncodingType_e

  Cyan4973

  committed
  477a010

  Copy full SHA for 477a010

• codemod: ZSTD_defaultPolicy_e -> ZSTD_DefaultPolicy_e

  Cyan4973

  committed
  0442e43

  Copy full SHA for 0442e43

• ZSTD_entropyCompressSeqStore_internal() can accept an externally defined literals buffer

  Cyan4973

  committed
  e9f8a11

  Copy full SHA for e9f8a11

• created ZSTD_entropyCompressSeqStore_wExtLitBuffer()

  Show description for 0165eeb
  Cyan4973

  committed
  0165eeb

  Copy full SHA for 0165eeb

• codemod: ZSTD_sequenceFormat_e -> ZSTD_SequenceFormat_e

  Show description for c97522f
  Cyan4973

  committed
  c97522f

  Copy full SHA for c97522f

• codemod: ZSTD_sequenceCopier -> ZSTD_SequenceCopier_f

  Cyan4973

  committed
  894ea31

  Copy full SHA for 894ea31

• minor: simplify ZSTD_selectSequenceCopier

  Cyan4973

  committed
  1ac79ba

  Copy full SHA for 1ac79ba

• scope: ZSTD_copySequencesToSeqStore*() are private to ZSTD_compress.c

  Show description for 76dd3a9
  Cyan4973

  committed
  76dd3a9

  Copy full SHA for 76dd3a9

• fix proper type for .forceNonContiguous

  Cyan4973

  committed
  03d95f9

  Copy full SHA for 03d95f9

• enable proper type

  Cyan4973

  committed
  5359d16

  Copy full SHA for 5359d16

• codemod: ZSTD_sequencePosition -> ZSTD_SequencePosition

  Cyan4973

  committed
  30671d7

  Copy full SHA for 30671d7

• codemod: ZSTD_buildSeqStore_e -> ZSTD_BuildSeqStore_e

  Cyan4973

  committed
  fa46894

  Copy full SHA for fa46894

• codemod: ZSTD_matchState_t -> ZSTD_MatchState_t

  Cyan4973

  committed
  5df80ac

  Copy full SHA for 5df80ac

• codemod: repcodes_t -> Repcodes_t

  Cyan4973

  committed
  41c667c

  Copy full SHA for 41c667c

• codemod: rawSeqStore_t -> RawSeqStore_t

  Cyan4973

  committed
  25bef24

  Copy full SHA for 25bef24

• codemod: ZSTD_blockCompressor -> ZSTD_BlockCompressor_f

  Cyan4973

  committed
  08edecb

  Copy full SHA for 08edecb

• doc: add mention of frame checksum incompatibility

  Show description for e0f3aae
  Cyan4973

  committed
  e0f3aae

  Copy full SHA for e0f3aae

• fix minor artifact error in single_file_lib

  Cyan4973

  committed
  2503b64

  Copy full SHA for 2503b64

• minor simplification

  Cyan4973

  committed
  13b9296

  Copy full SHA for 13b9296

• codemod: ZSTD_paramSwitch_e -> ZSTD_ParamSwitch_e

  Cyan4973

  committed
  56cfb78

  Copy full SHA for 56cfb78

• codemod: ZSTD_cParamMode_e -> ZSTD_CParamMode_e

  Cyan4973

  committed
  4ef9d7d

  Copy full SHA for 4ef9d7d

• ZSTD_SequenceCopier_f no returns the nb of bytes consumed from input

  Show description for 047db4f
  Cyan4973

  committed
  047db4f

  Copy full SHA for 047db4f

• minor: more accurate variable scope

  Cyan4973

  committed
  bcb1509

  Copy full SHA for bcb1509

• produced ZSTD_compressSequencesAndLiterals() as a separate pipeline

  Show description for 14a21e4
  Cyan4973

  committed
  14a21e4

  Copy full SHA for 14a21e4

• added unit tests to ZSTD_compressSequencesAndLiterals()

  Show description for 0b013b2
  Cyan4973

  committed
  0b013b2

  Copy full SHA for 0b013b2

• prepare fullbench for a refactor

  Cyan4973

  committed
  09964c6

  Copy full SHA for 09964c6

• refactor fullbench

  Show description for 4c097b4
  Cyan4973

  committed
  4c097b4

  Copy full SHA for 4c097b4

• minor: test reordering

  Cyan4973

  committed
  c540976

  Copy full SHA for c540976

• minor variable renaming

  Cyan4973

  committed
  8b7e1b7

  Copy full SHA for 8b7e1b7

• fullbench: preparation functions are now in charge of allocating buffers

  Cyan4973

  committed
  c050ae4

  Copy full SHA for c050ae4

• add the compressSequences() benchmark scenario

  Cyan4973

  committed
  8ab0409

  Copy full SHA for 8ab0409

• fullbench: switch default generator to lorem ipsum

  Show description for ac05ea8
  Cyan4973

  committed
  ac05ea8

  Copy full SHA for ac05ea8

• fullbench: new scenario: compressSequencesAndLiterals()

  Cyan4973

  committed
  f281497

  Copy full SHA for f281497

• minor conversion warning fix

  Cyan4973

  committed
  0a5c080

  Copy full SHA for 0a5c080

• minor optimization for ZSTD_compressSequencesAndLiterals()

  Show description for 1c8f5b0
  Cyan4973

  committed
  1c8f5b0

  Copy full SHA for 1c8f5b0

• minor doc update

  Cyan4973

  committed
  f176514

  Copy full SHA for f176514

• minor optimization: only track seqPos->posInSrc when validateSequences is enabled

  Show description for a288751
  Cyan4973

  committed
  a288751

  Copy full SHA for a288751

• optimization: instantiate specialized version without Sequence checking code

  Show description for 1f6d681
  Cyan4973

  committed
  1f6d681

  Copy full SHA for 1f6d681

• updated documentation on validateSequence

  Cyan4973

  committed
  d2d0fda

  Copy full SHA for d2d0fda

• minor: cleaner function parameter repcodeResolution

  Cyan4973

  committed
  ca8bd83

  Copy full SHA for ca8bd83

• change advanced parameter name: ZSTD_c_repcodeResolution

  Show description for 5164d44
  Cyan4973

  committed
  5164d44

  Copy full SHA for 5164d44

• ZSTD_compressSequencesAndLiterals() now supports multi-blocks frames.

  Cyan4973

  committed
  31b5ef2

  Copy full SHA for 31b5ef2

• added tests

  Show description for f0d0d95
  Cyan4973

  committed
  f0d0d95

  Copy full SHA for f0d0d95

• update Visual Studio solutions

  Cyan4973

  committed
  6f8c104

  Copy full SHA for 6f8c104

• removed fullbench-dll project from visual solutions

  Cyan4973

  committed
  47edd0a

  Copy full SHA for 47edd0a

• fixed incorrect assert

  Cyan4973

  committed
  f617e86

  Copy full SHA for f617e86

• attempt to silence Visual Studio warning about fopen()

  Cyan4973

  committed
  61ac831

  Copy full SHA for 61ac831

• change name to ZSTD_convertSequences*()

  Cyan4973

  committed
  d48e330

  Copy full SHA for d48e330

• added benchmark for ZSTD_convertBlockSequences_wBlockDelim()

  Cyan4973

  committed
  95ad9e4

  Copy full SHA for 95ad9e4

• improved speed of the Sequences converter

  Cyan4973

  committed
  12c47d3

  Copy full SHA for 12c47d3

• fixed minor conversion warning

  Cyan4973

  committed
  b7b4e86

  Copy full SHA for b7b4e86

• fixed minor error in one benchmark scenario

  Cyan4973

  committed
  ad023b3

  Copy full SHA for ad023b3

• ZSTD_compressSequencesAndLiterals requires srcSize as parameter

  Show description for 0a54f6f
  Cyan4973

  committed
  0a54f6f

  Copy full SHA for 0a54f6f

• added a test for ZSTD_compressSequencesAndLiterals

  Show description for a80f55f
  Cyan4973

  committed
  a80f55f

  Copy full SHA for a80f55f

• add dedicated error code for special case

  Show description for b339eff
  Cyan4973

  committed
  b339eff

  Copy full SHA for b339eff

• ensure that srcSize is controlled

  Cyan4973

  committed
  ab0f179

  Copy full SHA for ab0f179

• fixed minor error in preparation of one fullbench scenario

  Cyan4973

  committed
  52a9bc6

  Copy full SHA for 52a9bc6

• add a check, to return an error if Sequence validation is enabled

  Show description for 76445bb
  Cyan4973

  committed
  76445bb

  Copy full SHA for 76445bb

• added parameter litCapacity

  Show description for b7a9e69
  Cyan4973

  committed
  b7a9e69

  Copy full SHA for b7a9e69

• minor: use MEM_writeLE24()

  Show description for 522adc3
  Cyan4973

  committed
  522adc3

  Copy full SHA for 522adc3

• restore invocation of ZSTD_entropyCompressSeqStore()

  Show description for 47cbfc8
  Cyan4973

  committed
  47cbfc8

  Copy full SHA for 47cbfc8

Commits on Dec 24, 2024

• added fuzzer test for compressSequencesAndLiterals()

  Show description for f8725e8
  Cyan4973

  committed
  f8725e8

  Copy full SHA for f8725e8

• add one valid test case

  Show description for 7b294ca
  Cyan4973

  committed
  7b294ca

  Copy full SHA for 7b294ca

• fixed another invalid scenario

  Show description for 72ce56b
  Cyan4973

  committed
  72ce56b

  Copy full SHA for 72ce56b