Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Filters
























178 Hits in 5.2 sec

Multi-level Forwarding and Scheduling Recovery Algorithm in Rapidly-changing Network for Erasure-coded Clusters [article]

Hai Zhou, Dan Feng, Yuchong Hu
2021 arXiv   pre-print

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (5.5 MB)
https://web.archive.org/web/20201106122000/https://arxiv.org/ftp/arxiv/papers/2011/2011.01410.pdf

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL. The file type is application/pdf.

Note that this fulltext copy is not of the "primary" version of this work. The version it corresponds to is:

2020 pre-print arXiv:2011.01410v1 fatcat:ocbf4vcb2vh7pa4i4kosrijbda
A key design goal of erasure-coded clusters is to reduce the repair time. The existing Erasure-coded data repair schemes are roughly classified into two categories: 1.  ...  However, these solutions are difficult to cope with the heterogeneous and Rapidly-changing network in erasure-coded clusters.  ...  MOTIVATION Fig. 3 Bandwidth-aware multi-level forwarding repair We first observe a three-node Erasure-code cluster, node1, node2 and node3.  ... 
arXiv:2011.01410v2 fatcat:cawawgob2vglbbp22ymvd5h7hi
Multiple Versions
Citation

Hai Zhou, Dan Feng, Yuchong Hu. "Multi-level Forwarding and Scheduling Recovery Algorithm in Rapidly-changing Network for Erasure-coded Clusters." arXiv (2021)

Rack Aware Data Placement for Network Consumption in Erasure-Coded Clustered Storage Systems

Bilin Shao, Dan Song, Genqing Bian, Yu Zhao
2018 Information  

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (3.0 MB)
https://web.archive.org/web/20190505214014/https://res.mdpi.com/information/information-09-00150/article_deploy/information-09-00150.pdf?filename=&attachment=1

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2019; you can also visit the original URL. The file type is application/pdf.

The amount of encoded data replication in an erasure-coded clustered storage system has a great impact on the bandwidth consumption and network latency, mostly during data reconstruction.  ...  Theoretical analysis and simulation results show that our proposed strategy greatly reduces the frequency and data volume of the cross-rack transmission during data reconstruction.  ...  The Architecture of the Erasure-Coded Clustered System In the erasure-coded cluster storage system, data blocks and encoding blocks are stored separately in different nodes of different racks.  ... 
doi:10.3390/info9070150 fatcat:pwufr6ulfvfevion5vyrwpyhda
DOAJ Szczepanski
Citation

Bilin Shao, Dan Song, Genqing Bian, Yu Zhao. "Rack Aware Data Placement for Network Consumption in Erasure-Coded Clustered Storage Systems." Information 9.7 (2018) 150

Exploiting Redundancies and Deferred Writes to Conserve Energy in Erasure-Coded Storage Clusters

Jianzhong Huang, Fenghao Zhang, Xiao Qin, Changsheng Xie
2013 ACM Transactions on Storage  

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (2.0 MB)
https://web.archive.org/web/20170921233739/http://www.eng.auburn.edu/%7Exqin/pubs/huang_tos13.pdf

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL. The file type is application/pdf.

The experimental results indicate that compared to energy-oblivious erasure-coded storage, ECS 2 can save the energy used by storage clusters up to 29.8% and 28.0% in read-intensive and write-dominated  ...  We present a power-efficient scheme for erasure-coded storage clusters-ECS 2 -which aims to offer high energy efficiency with marginal reliability degradation.  ...  The authors also appreciate Shuaijun Han and Shenjie Cui for their assistance in the development of the experimental platform for this study.  ... 
doi:10.1145/2491472.2491473 fatcat:uehiywrkpbajlcjszymx3ywwpm
Citation

Jianzhong Huang, Fenghao Zhang, Xiao Qin, Changsheng Xie. "Exploiting Redundancies and Deferred Writes to Conserve Energy in Erasure-Coded Storage Clusters." ACM Transactions on Storage 9.2 (2013) 1-29

Repair Pipelining for Erasure-Coded Storage: Algorithms and Evaluation [article]

Xiaolu Li, Zuoru Yang, Jinhong Li, Runhui Li, Patrick P. C. Lee, Qun Huang, Yuchong Hu
2020 arXiv   pre-print

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (753.8 kB)
https://web.archive.org/web/20201124042101/https://arxiv.org/pdf/1908.01527v3.pdf

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL. The file type is application/pdf.

Other Versions

2020 pre-print
arXiv:1908.01527v2 fatcat:xscmdpfhlfeavcvetud7wp2w6m
2019 pre-print
arXiv:1908.01527v1 fatcat:qqul7qzae5e4fepe4vc7hytbl4
We propose repair pipelining, a technique that speeds up the repair performance in general erasure-coded storage.  ...  By carefully scheduling the repair of failed data in small-size units across storage nodes in a pipelined manner, repair pipelining reduces the single-block repair time to approximately the same as the  ...  Conclusions Repair pipelining is a general technique that reduces the single-block repair time to almost the same as the normal read time for a single available block in erasure-coded storage.  ... 
arXiv:1908.01527v3 fatcat:poj7lo37azd3nnu4u3i7qzjdie
Multiple Versions
Citation

Xiaolu Li, Zuoru Yang, Jinhong Li, Runhui Li, Patrick P. C. Lee, Qun Huang, Yuchong Hu. "Repair Pipelining for Erasure-Coded Storage: Algorithms and Evaluation." arXiv (2020)

Boosting Full-Node Repair in Erasure-Coded Storage

Shiyao Lin, Guowen Gong, Zhirong Shen, Patrick P. C. Lee, Jiwu Shu
2021 USENIX Annual Technical Conference  

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (1.2 MB)
https://web.archive.org/web/20220803233440/https://www.usenix.org/system/files/atc21-lin.pdf

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2022; you can also visit the original URL. The file type is application/pdf.

A variety of erasure codes and repair algorithms are designed in recent years to relieve the repair traffic, yet we unveil via careful analysis that they are still plagued by several limitations, which  ...  We present RepairBoost, a scheduling framework that can assist existing linear erasure codes and repair algorithms to boost the full-node repair performance.  ...  Acknowledgments We thank our shepherd, Mike Mesnier, and the anonymous reviewers for the comments on our camera-ready preparations.  ... 
dblp:conf/usenix/LinGSLS21 fatcat:xbq7m7brtrfntfk76tsy7d7ks4
Citation

Shiyao Lin, Guowen Gong, Zhirong Shen, Patrick P. C. Lee, Jiwu Shu. "Boosting Full-Node Repair in Erasure-Coded Storage." USENIX Annual Technical Conference (2021) 641-655

The CORE Storage Primitive: Cross-Object Redundancy for Efficient Data Repair & Access in Erasure Coded Storage [article]

Kyumars Sheykh Esmaili, Lluis Pamies-Juarez, Anwitaman Datta
2013 arXiv   pre-print

Preserved Fulltext

File Archive PDF (888.6 kB)
https://archive.org/download/arxiv-1302.5192/1302.5192.pdf

The Internet Archive has a preservation copy of this work in our general collections. The file type is application/pdf.

Other Versions

2013 pre-print
arXiv:1302.5192v1 fatcat:hnjuwhuti5hovmqzx3svlo6n4a
2013 pre-print
arXiv:1302.5192v2 fatcat:szjoag3hrfe6hiy3zi467wh3ee
2013 pre-print
arXiv:1302.5192v3 fatcat:kja5rseghjbrbnkomoo7qeoxqi
We build upon this study to design CORE, a general storage primitive that we integrate into HDFS. We benchmark this implementation in a proprietary cluster and in EC2.  ...  In this paper, we take an engineering alternative, exploring the use of simple and mature techniques -juxtaposing a standard erasure code with RAID-4 like parity.  ...  codes [1, 3, 23] , applying network coding techniques [2, 4, 20] , or designing completely new codes with lower repair fan-in [13] [14] [15] , and (iii) fast creation of erasure coded redundancy [  ... 
arXiv:1302.5192v4 fatcat:rwljkazcc5azfi7khqpsy7vc5i
Multiple Versions
Citation

Kyumars Sheykh Esmaili, Lluis Pamies-Juarez, Anwitaman Datta. "The CORE Storage Primitive: Cross-Object Redundancy for Efficient Data Repair & Access in Erasure Coded Storage." arXiv (2013)

RAFI: Risk-Aware Failure Identification to Improve the RAS in Erasure-coded Data Centers

Juntao Fang, Shenggang Wan, Xubin He
2018 USENIX Annual Technical Conference  

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (1.0 MB)
https://web.archive.org/web/20210803013903/https://www.usenix.org/system/files/conference/atc18/atc18-fang.pdf

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2021; you can also visit the original URL. The file type is application/pdf.

Data reliability and availability, and serviceability (RAS) of erasure-coded data centers are highly affected by data repair induced by node failures.  ...  Compared to the recovery phase of the data repair, which is widely studied and well optimized, the failure identification phase of the data repair is less investigated.  ...  We are grateful to anonymous reviewers and particularly our shepherd Dahlia Malkhi for their helpful comments and suggestions.  ... 
dblp:conf/usenix/FangWH18 fatcat:nvp7jnwksraevgleilj6ocoykq
Citation

Juntao Fang, Shenggang Wan, Xubin He. "RAFI: Risk-Aware Failure Identification to Improve the RAS in Erasure-coded Data Centers." USENIX Annual Technical Conference (2018) 495-506

S3: Joint Scheduling and Source Selection for Background Traffic in Erasure-Coded Storage

Shijing Li, Tian Lan, Moo-Ryong Ra, Rajesh Panta
2017 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS)  

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (641.8 kB)
https://web.archive.org/web/20170907082114/http://www2.seas.gwu.edu/~tlan/papers/S3_ICDCS_2017.pdf

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL. The file type is application/pdf.

In this paper, we claim that an optimal scheduling algorithm that aims to maximize the number of background tasks completed before deadlines must simultaneously consider deadline-aware scheduling, network  ...  Erasure-coded storage systems have gained considerable adoption recently since they can provide the same level of reliability with significantly lower storage overhead compared to replicated systems.  ...  LPST ALGORITHM DESIGN AND COMPETITORS In this section, we describe our proposed algorithm, called Linear Programming for Selected Tasks (LPST), which harnesses resource-aware chunk selection, deadline-aware  ... 
doi:10.1109/icdcs.2017.204 dblp:conf/icdcs/LiLRP17 fatcat:4v7wtqs2incz5epusdhlgwimpy
Citation

Shijing Li, Tian Lan, Moo-Ryong Ra, Rajesh Panta. "S3: Joint Scheduling and Source Selection for Background Traffic in Erasure-Coded Storage." 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS) (2017) 2025-2030

Dominoes: Speculative Repair in Erasure-Coded Hadoop System

Xi Yang, Chen Feng, Zhiwei Xu, Xian-He Sun
2015 2015 IEEE 22nd International Conference on High Performance Computing (HiPC)  

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (1.5 MB)
https://web.archive.org/web/20170830065740/http://www.cs.iit.edu/~scs/psfiles/dominoes.pdf

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL. The file type is application/pdf.

In an erasure-coded system, data reconstruction time will be paid while tasks access the missing blocks during MapReduce job processing.  ...  Erasure coding can provide equivalent three-way fault tolerance to HDFS's default three replication mechanism but degrades data availability for task scheduling.  ...  This research is also supported in part by NSF under NSF grants CNS-0751200, CNS-1162540, and CNS-1526887.  ... 
doi:10.1109/hipc.2015.39 dblp:conf/hipc/YangFXS15 fatcat:k66mwepd5nh3dd5l5pffueil24
Citation

Xi Yang, Chen Feng, Zhiwei Xu, Xian-He Sun. "Dominoes: Speculative Repair in Erasure-Coded Hadoop System." 2015 IEEE 22nd International Conference on High Performance Computing (HiPC) (2015) 366-375

Fatman

An Qin, Dianming Hu, Jun Liu, Wenjun Yang, Dai Tan
2014 Proceedings of the VLDB Endowment  

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (426.5 kB)
https://web.archive.org/web/20160129045451/http://www.vldb.org/pvldb/vol7/p1748-baidu.pdf

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2016; you can also visit the original URL. The file type is application/pdf.

Fatman is specifically designed for enhancing the utilization of existing storage resources and cutting down the hardware purchase cost.  ...  It firstly improves data reliability by applying disk failure prediction to minish failure recovery cost, named fault-aware data management, dramatically reduces the MTTR by 76.3% and decreases file crash  ...  The integration of failure prediction and erasure codes is never reported in recent articles, especially applying prediction to pre-schedule decoding.  ... 
doi:10.14778/2733004.2733078 fatcat:7pnkj34lgnfhxcqssoxgq4iq4i
Citation

An Qin, Dianming Hu, Jun Liu, Wenjun Yang, Dai Tan. "Fatman." Proceedings of the VLDB Endowment 7.13 (2014) 1748-1753

TOWARDS DIGITAL TWINS FOR OPTIMIZING METRICS IN DISTRIBUTED STORAGE SYSTEMS - A REVIEW

May Itani, Layal Abu Daher, Ahmad Hammoud
2023 BAU Journal - Science and Technology  

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (231.6 kB)
https://web.archive.org/web/20240112221117/https://digitalcommons.bau.edu.lb/cgi/viewcontent.cgi?article=1118&context=stjournal

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2024; you can also visit the original URL. The file type is application/pdf.

These constitute: Repair bandwidth, load imbalance, latency, disk failure, and reliability. • Repair Bandwidth is the transmission efficiency during the recovery process in DSSs, is defined as the average  ...  The paper describes the design method, numerical analysis, simulation study, and experimental validation of this method.  ... 
doi:10.54729/2959-331x.1118 fatcat:3bdrygiebrg2hivh2cu662oz2q
Citation

May Itani, Layal Abu Daher, Ahmad Hammoud. "TOWARDS DIGITAL TWINS FOR OPTIMIZING METRICS IN DISTRIBUTED STORAGE SYSTEMS - A REVIEW." BAU Journal - Science and Technology 5.1 (2023)

XORing elephants

Maheswaran Sathiamoorthy, Megasthenis Asteris, Dimitris Papailiopoulos, Alexandros G. Dimakis, Ramkumar Vadali, Scott Chen, Dhruba Borthakur
2013 Proceedings of the VLDB Endowment  

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (905.5 kB)
https://web.archive.org/web/20190303134412/http://smahesh.com:80/HadoopUSC/Xorbas.pdf

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2019; you can also visit the original URL. The file type is application/pdf.

We present a novel family of erasure codes that are efficiently repairable and offer higher reliability compared to Reed-Solomon codes.  ...  Reed-Solomon codes are the standard design choice and their high repair cost is often considered an unavoidable price to pay for high storage efficiency and high reliability.  ...  in repair bandwidth and disk I/O.  ... 
doi:10.14778/2535573.2488339 fatcat:ncei5mlrfjhjvgndoj463tqv6e
Citation

Maheswaran Sathiamoorthy, Megasthenis Asteris, Dimitris Papailiopoulos, Alexandros G. Dimakis, Ramkumar Vadali, Scott Chen, Dhruba Borthakur. "XORing elephants." Proceedings of the VLDB Endowment 6.5 (2013) 325-336

Fault tolerance in big data storage and processing systems: A review on challenges and solutions

Muntadher Saadoon, Siti Hafizah Ab. Hamid, Hazrina Sofian, Hamza H.M. Altarturi, Zati Hakim Azizul, Nur Nasuha
2021 Ain Shams Engineering Journal  

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (1.2 MB)
https://web.archive.org/web/20210719225231/https://pdf.sciencedirectassets.com/270550/AIP/1-s2.0-S2090447921002896/main.pdf?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEJ7%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIEtLgGNjouGL2DHb6oz8af%2F8IGuloX%2FRtGTnbLUSOzuFAiEA3Q142Wjs7dSGzIFWQTFFx5avroCCCRuhfLkWSAclrk4qgwQIl%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAEGgwwNTkwMDM1NDY4NjUiDFukb2DnK6sYuReyDCrXA5pztDe3mfFZSEJvvjUK3uJQo7e9F6aFBgA3QUbFtytY3LCa6t4eu1goebE31ZbRVQaAf7tysOrDZo6Fft5NzuyJlpzjMxnX1wv15O0Di8Zem14j4I0djyO%2BIkWvnUFZXBcLWludDyolVsgNwWJRCCT2TvTHR3YgOlJkqAZFje3nXdPrNQjWVyKLACuYq0oFTLznuHTelYBPi6DGU650fRKAjRnpij6Vy0upxq76z7wHWNfYjpOYg9lEjUn29jUuAZ%2FaUadMV2fR6Wm2%2BU%2BW%2BuHLhlKeHIulh5t6JFXimQFQl9ZgSGOmxem8RxT9Bna4vgKAgM76Egr1nPa3FDtDpTNI3WpNQiaK0wMyoK%2B5yjG%2FVJx8%2Bb3gQtdAe2uahHeAd1zvrYF0gCDO7GuuI%2Fa94Px6HiROz3BDaDvPmdhMdDJSvY36UUKu%2BecZdfNQ8kdL4AhFp006w1vV2La2jkQBQIxDN88JJAKDF6rTg7sH43YvNBq0uU%2FraVTmMTq9BubTFNmHnvSEhyPMxpgBV%2BPs0laYLzQOBUJ8kBqjJIO9EnqA2bYRkUWNRa6oGkLuKdD949eBy3Xge5JgTcqSHfqBqFTM57%2FXmvpbolNn3P0wXnP3%2BpsyLg1D8jDO6NeHBjqlASrzmzY9HOx2DYuEHbHQEAsAo1VCqXmkTQHvZM39ysq4JgA0sQMSoFq8TT9Yc5mkO70VjL1qH32Q%2F8ErYS7IF62rbC44S6gHT3iICr3eMSSBzmt7D6aP2p%2F%2BV02M4IYSHfRZ4N7CfIgi5syfYHHuEWOTKkgLQPIjksGVXxDogbOM5cjGXh18bwZKvYE94z4oPDJcGqon8rqmS4bvbk3FBihB0%2BuwIA%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210719T225227Z&X-Amz-SignedHeaders=host&X-Amz-Expires=300&X-Amz-Credential=ASIAQ3PHCVTYTLZMPNNA%2F20210719%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=0902de2e245a3fdd17d68e317da996fc6b21c3178f64c8485673154b7c9f1870&hash=80d93be221620dd289aff1aa6fad70e717523e13a747305444303e31f81f408d&host=68042c943591013ac2b2430a89b270f6af2c76d8dfd086a07176afe7c76c2c61&pii=S2090447921002896&tid=spdf-a6d5cfb1-e902-4f3b-8367-1d9300035933&sid=cbebe789906e134a2879d032c5f68b04dba9gxrqa&type=client

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2021; you can also visit the original URL. The file type is application/pdf.

This study aims to provide a consistent understanding of fault tolerance in big data systems and highlights common challenges that hinder the improvement in fault tolerance efficiency.  ...  Achieving an efficient fault tolerance solution in a big data system is challenging because fault tolerance must meet some constraints related to the system performance and resource consumption.  ...  A hybrid technique that involves erasure coding and replication can also be applied in the storage systems.  ... 
doi:10.1016/j.asej.2021.06.024 fatcat:bk7hhhss7fe73ok24df6tjrfum
DOAJ
Citation

Muntadher Saadoon, Siti Hafizah Ab. Hamid, Hazrina Sofian, Hamza H.M. Altarturi, Zati Hakim Azizul, Nur Nasuha. "Fault tolerance in big data storage and processing systems: A review on challenges and solutions." Ain Shams Engineering Journal (2021)

Erasure Coding-oriented Data Update for Cloud Storage: A Survey

Yifei Xiao, Shijie Zhou, Linpeng Zhong
2020 IEEE Access  

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (3.5 MB)
https://web.archive.org/web/20201025140928/https://ieeexplore.ieee.org/ielx7/6287639/6514899/09235567.pdf?tp=&arnumber=9235567&isnumber=6514899&ref=

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL. The file type is application/pdf.

We also appreciate Jingwei Li and Hu Xiong for their sincere help.  ...  an encoding matrix G ∈ C, thus, the repair bandwidth per symbol of repaired data for the proposed code with IA tends to n−1 n−kn .  ...  To improve DR efficiency, the main body of the literature on erasure coding has made great contributions on designing excellent erasure codes or improving existing erasure codes [9] , [27] , [29] ,  ... 
doi:10.1109/access.2020.3033024 fatcat:5lddrtxervaxle6eqx2koioh24
DOAJ
Citation

Yifei Xiao, Shijie Zhou, Linpeng Zhong. "Erasure Coding-oriented Data Update for Cloud Storage: A Survey." IEEE Access (2020) 1-1

XORing Elephants: Novel Erasure Codes for Big Data [article]

Maheswaran Sathiamoorthy, Megasthenis Asteris, Dimitris Papailiopoulos, Alexandros G. Dimakis, Ramkumar Vadali, Scott Chen, Dhruba Borthakur
2013 arXiv   pre-print

Preserved Fulltext

File Archive PDF (1.1 MB)
https://archive.org/download/arxiv-1301.3791/1301.3791.pdf

The Internet Archive has a preservation copy of this work in our general collections. The file type is application/pdf.

We present a novel family of erasure codes that are efficiently repairable and offer higher reliability compared to Reed-Solomon codes.  ...  Reed-Solomon codes are the standard design choice and their high repair cost is often considered an unavoidable price to pay for high storage efficiency and high reliability.  ...  in repair bandwidth and disk I/O.  ... 
arXiv:1301.3791v1 fatcat:7u5w67cebbbm3eumjdxm2vhpje
Citation

Maheswaran Sathiamoorthy, Megasthenis Asteris, Dimitris Papailiopoulos, Alexandros G. Dimakis, Ramkumar Vadali, Scott Chen, Dhruba Borthakur. "XORing Elephants: Novel Erasure Codes for Big Data." arXiv (2013)

« Previous Showing results 1 — 15 out of 178 results