Controlling the chunk-size in deduplication systems

Michael Hirsch; Shmuel T. Klein; Dana Shapira; Yair Toaff

Controlling the chunk-size in deduplication systems

Michael Hirsch, Shmuel T. Klein, Dana Shapira, Yair Toaff

Department of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

A special case of data compression in which repeated chunks of data are stored only once is known as deduplication. The input data is cut into chunks and a cryptographically strong hash value of each (different) chunk is stored. To restrict the influence of small inserts and deletes to local perturbations, the chunk boundaries are usually defined in a data dependent way, which implies that the chunks are of variable length. Usually, the chunk sizes may spread over a large range, which may have a negative impact on the storage performance. This may be dealt with by imposing artificial lower and upper bounds. This paper suggests an alternative by which the chunk size distribution is controlled in a natural way. Some analytical and experimental results are given.

Original language	English
Title of host publication	Proceedings of the Prague Stringology Conference 2015, PSC 2015
Editors	Jan Zd'arek, Jan Holub
Pages	78-89
Number of pages	12
ISBN (Electronic)	9788001057872
State	Published - 2015
Event	19th Prague Stringology Conference, PSC 2015 - Prague, Czech Republic Duration: 24 Aug 2015 → 26 Aug 2015

Publication series

Name	Proceedings of the Prague Stringology Conference 2015, PSC 2015

Conference

Conference	19th Prague Stringology Conference, PSC 2015
Country/Territory	Czech Republic
City	Prague
Period	24/08/15 → 26/08/15

Cite this

@inproceedings{528d1df68ebc4582b345f1c9074aa51c,

title = "Controlling the chunk-size in deduplication systems",

abstract = "A special case of data compression in which repeated chunks of data are stored only once is known as deduplication. The input data is cut into chunks and a cryptographically strong hash value of each (different) chunk is stored. To restrict the influence of small inserts and deletes to local perturbations, the chunk boundaries are usually defined in a data dependent way, which implies that the chunks are of variable length. Usually, the chunk sizes may spread over a large range, which may have a negative impact on the storage performance. This may be dealt with by imposing artificial lower and upper bounds. This paper suggests an alternative by which the chunk size distribution is controlled in a natural way. Some analytical and experimental results are given.",

author = "Michael Hirsch and Klein, {Shmuel T.} and Dana Shapira and Yair Toaff",

note = "Publisher Copyright: {\textcopyright} Czech Technical University in Prague, Czech Republic.; 19th Prague Stringology Conference, PSC 2015 ; Conference date: 24-08-2015 Through 26-08-2015",

year = "2015",

language = "אנגלית",

series = "Proceedings of the Prague Stringology Conference 2015, PSC 2015",

pages = "78--89",

editor = "Jan Zd'arek and Jan Holub",

booktitle = "Proceedings of the Prague Stringology Conference 2015, PSC 2015",

}

Controlling the chunk-size in deduplication systems. / Hirsch, Michael; Klein, Shmuel T.; Shapira, Dana et al.
Proceedings of the Prague Stringology Conference 2015, PSC 2015. ed. / Jan Zd'arek; Jan Holub. 2015. p. 78-89 (Proceedings of the Prague Stringology Conference 2015, PSC 2015).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Controlling the chunk-size in deduplication systems

AU - Hirsch, Michael

AU - Klein, Shmuel T.

AU - Shapira, Dana

AU - Toaff, Yair

PY - 2015

Y1 - 2015

N2 - A special case of data compression in which repeated chunks of data are stored only once is known as deduplication. The input data is cut into chunks and a cryptographically strong hash value of each (different) chunk is stored. To restrict the influence of small inserts and deletes to local perturbations, the chunk boundaries are usually defined in a data dependent way, which implies that the chunks are of variable length. Usually, the chunk sizes may spread over a large range, which may have a negative impact on the storage performance. This may be dealt with by imposing artificial lower and upper bounds. This paper suggests an alternative by which the chunk size distribution is controlled in a natural way. Some analytical and experimental results are given.

AB - A special case of data compression in which repeated chunks of data are stored only once is known as deduplication. The input data is cut into chunks and a cryptographically strong hash value of each (different) chunk is stored. To restrict the influence of small inserts and deletes to local perturbations, the chunk boundaries are usually defined in a data dependent way, which implies that the chunks are of variable length. Usually, the chunk sizes may spread over a large range, which may have a negative impact on the storage performance. This may be dealt with by imposing artificial lower and upper bounds. This paper suggests an alternative by which the chunk size distribution is controlled in a natural way. Some analytical and experimental results are given.

UR - http://www.scopus.com/inward/record.url?scp=84978493601&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:84978493601

T3 - Proceedings of the Prague Stringology Conference 2015, PSC 2015

SP - 78

EP - 89

BT - Proceedings of the Prague Stringology Conference 2015, PSC 2015

A2 - Zd'arek, Jan

A2 - Holub, Jan

T2 - 19th Prague Stringology Conference, PSC 2015

Y2 - 24 August 2015 through 26 August 2015

ER -

Controlling the chunk-size in deduplication systems

Abstract

Publication series

Conference

Other files and links

Fingerprint

Cite this