一种纠删码条带化数据的一致性检查方法

doi:10.16183/j.cnki.jsjtu.2024.035

上海交通大学学报 ›› 2024, Vol. 58 ›› Issue (4): 579-584.doi: 10.16183/j.cnki.jsjtu.2024.035

• 电子信息与电气工程 • 上一篇

一种纠删码条带化数据的一致性检查方法

许亮业¹, 石连星²(), 单蓉胜³

1.上海交通大学医学院附属上海儿童医学中心,上海 200127
2.上海霄云信息科技有限公司,上海 200240
3.上海交通大学电子信息与电气工程学院,上海200240

收稿日期:2024-01-24 修回日期:2024-02-20 接受日期:2024-02-22 出版日期:2024-04-28 发布日期:2024-04-30
通讯作者: 石连星,硕士;E-mail:shilianxing@shxiaoyun.com.cn.
作者简介:许亮业(1984-),正高级工程师,从事医疗信息化研究.

A Consistency Checking Method for Erasure-Coded Striped Data

XU Liangye¹, SHI Lianxing²(), SHAN Rongsheng³

1. Shanghai Children’s Medical Center, Shanghai Jiao Tong University School of Medicine, Shanghai 200127, China
2. Shanghai Xiaoyun Info Tech Co., Ltd., Shanghai 200240, China
3. School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China

Received:2024-01-24 Revised:2024-02-20 Accepted:2024-02-22 Online:2024-04-28 Published:2024-04-30

摘要/Abstract

摘要：

纠删码冗余策略常用于分布式存储系统.在纠删码数据中,条带是一致性检查的基本单元,每个条带包含多份原始数据单元和校验数据单元.为了减少纠删码条带化数据一致性检查的读取开销,提高纠删码数据一致性检查和读后写的效率,在执行纠删码条带化数据写入时,为每个条带单元加入自修正数据标签 (SCDT),后续对每个条带的一致性检查基于SCDT完成.该方法不需要读取每个条带中所有数据单元即可完成该条带的一致性检查,将一致性检查效率提升了1.7~2.6倍,并且当写入数据更新的条带单元数小于临界值时,可以有效减少写入的输入输出(IO)交互次数.本文方法可以更好地应对条带化数据组的部分更新,同时提高一致性检查效率.

关键词: 分布式存储系统, 条带化数据, 一致性检查, 数据标签, 纠删码

Abstract:

Erasure code is commonly used in distributed storage systems. The stripe is the basic unit of consistency check in erasure-coded data, including multiple original stripe units and verification stripe units. In order to reduce the cost of reading for consistency check of erasure-coded striped data and improve the efficiency of erasure-coded data consistency check and reading-after-writing, self-correction data tags (SCDTs) is added to each stripe unit when writing erasure-coded data in striping mode, based on which, the consistency checks of each stripe are implemented. The method proposed can complete the consistency check of a stripe without reading all data units in the stripe, which improves the efficiency of consistency checks by 1.7 to 2.6 times. Moreover, when the number of stripe units updated by written data is less than the critical value, it can effectively reduce the number of Input/Output (IO) interactions for writing. The method proposed can better handle partial updates of striped data sets while improving the efficiency of consistency checks.

Key words: distributed storage system, striped data, consistency check, data tag, erasure code

中图分类号:

TP391

许亮业, 石连星, 单蓉胜. 一种纠删码条带化数据的一致性检查方法[J]. 上海交通大学学报, 2024, 58(4): 579-584.

XU Liangye, SHI Lianxing, SHAN Rongsheng. A Consistency Checking Method for Erasure-Coded Striped Data[J]. Journal of Shanghai Jiao Tong University, 2024, 58(4): 579-584.

图/表 3

参考文献 21

[1]	艾瑞咨询. 2022年中国医疗信息化行业研究报告[EB/OL]. (2022-04-19) [2023-10-30]. https://www.thepaper.cn/newsDetail_forward_17686922 .
	iResearch Consulting Group. 2022 China Medical Informatization Industry Research Report[EB/OL]. (2022-04-19) [2023-10-30]. https://www.thepaper.cn/newsDetail_forward_17686922 .
[2]	HURLEN P, ØSTBYE T, BORTHNE A, et al. Introducing PACS to the late majority. A longitudinal study[J]. Journal of Digital Imaging, 2010, 23(1): 87-94. doi: 10.1007/s10278-008-9160-x pmid: 18979133
[3]	DESHMUKH V, SVB L, KULKARNI M, et al. PACS: An overview of the technology and related issues[J]. International Journal of Engineering Technology Science and Research, 2018, 5(5): 122-128.
[4]	GHEMAWAT S, GOBIOFF H, LEUNG S T. The Google file system[J]. ACM SIGOPS Operating Systems Review, 2003, 37(5): 29-43. doi: 10.1145/1165389.945450 URL
[5]	SHVACHKO K, KUANG H R, RADIA S, et al. The hadoop distributed file system[C]// Proceedings of the 26th Symposium on Mass Storage Systems and Technologies. Washington, USA: IEEE, 2010: 1-10.
[6]	WEIL S A, BRANDT S A, MILLER E L, et al. Ceph:A scalable, high-performance distributed file system[C]// Proceedings of the 7th Symposium on Operating Systems Design and Implementation. Seattle, USA: ACM, 2006: 307-320.
[7]	ADYA A, BOLOSKY W J, CASTRO M, et al. Farsite: Federated, available, and reliable storage for an incompletely trusted environment[J]. ACM SIGOPS Operating Systems Review, 2002, 36(1): 1-14.
[8]	REED I S, SOLOMON G. Polynomial codes over certain finite fields[J]. Journal of the Society for Industrial and Applied Mathematics, 1960, 8(2): 300-304. doi: 10.1137/0108018 URL
[9]	MURALIDHAR S, LLOYD W, ROY S, et al.f4: Facebook’s warm BLOB storage system[C]// Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation. Broomfield, USA: ACM, 2014: 383-398.
[10]	CALDER B, WANG J, OGUS A, et al. Windows Azure Storage: A highly available cloud storage service with strong consistency[C]// Proceedings of the 23rd ACM Symposium on Operating Systems Principles. Cascais Portugal: ACM, 2011: 143-157.
[11]	HUANG C, SIMITCI H, XU Y K, et al. Erasure coding in windows azure storage[C]// Proceedings of the 2012 USENIX Conference on Annual Technical Conference. Boston, USA: ACM, 2012: 15-26.
[12]	BERMUDEZ I, TRAVERSO S, MELLIA M, et al. Exploring the cloud from passive measurements: The Amazon AWS case[C]// Proceedings of 2013 IEEE INFOCOM. Turin, Italy: IEEE, 2013: 230-234.
[13]	KUBIATOWICZ J, BINDEL D, CHEN Y, et al. OceanStore: An architecture for global-scale persistent storage[J]. ACM SIGPLAN Notices, 2000, 35(11): 190-201. doi: 10.1145/356989.357007 URL
[14]	杨传辉. 大规模分布式存储系统: 原理解析与架构实战[M]. 北京: 机械工业出版社, 2013.
	YANG Chuanhui. Large-scale distributed storage system: Principles and architectures[M]. Beijing: China Machine Press, 2013.
[15]	Swift Team. Erasure code support[EB/OL]. (2019-08-14) [2024-02-23]. https://docs.openstack.org/swift/latest/overview_erasure_code.html .
[16]	BREWER E A. Towards robust distributed systems (abstract)[C]// Proceedings of the nineteenth annual ACM symposium on Principles of distributed computing. New York, USA: ACM, 2000: 7.
[17]	田俊峰, 王彦骉, 何欣枫, 等. 数据因果一致性研究综述[J]. 通信学报, 2020, 41(3): 154-167. doi: 10.11959/j.issn.1000-436x.2020055
	TIAN Junfeng, WANG Yanbiao, HE Xinfeng, et al. Survey on the causal consistency of data[J]. Journal on Communications, 2020, 41(3): 154-167. doi: 10.11959/j.issn.1000-436x.2020055
[18]	LAMPORT L. The part-time parliament[J]. ACM Transactions on Computer Systems, 1998, 116(2): 133-169.
[19]	LAMPORT L. Paxos made simple[J]. ACM SIGACT News, 2001, 32(4): 51-58.
[20]	ONGARO D, OUSTERHOUT J. In search of an understandable consensus algorithm[C]// Proceedings of the 2014 USENIX Conference on USENIX Annual Technical Conference. Philadelphia, USA: ACM, 2014: 305-320.
[21]	刘爱贵, 李纲彬, 阮薛平. 一种纠删码数据一致性保障方法及系统: CN 114064346 A[P]. 2022-02-18 [2023-10-28].
	LIU Aigui, LI Gangbin, RUAN Xueping. A method and system for ensuring consistency of erasure coded data: CN 114064346 A[P]. 2022-02-18 [2023-10-28].

一种纠删码条带化数据的一致性检查方法

A Consistency Checking Method for Erasure-Coded Striped Data

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 3

参考文献 21

相关文章 15

编辑推荐

Metrics

本文评价

[1]	黄鹤, 胡凯益, 杨澜, 王浩, 高涛, 王会峰. 智能行车记录仪图像去雾系统的FPGA设计[J]. 上海交通大学学报, 2024, 58(4): 565-578.
[2]	邢志伟, 阚犇, 刘子硕, 李彪, 罗谦. 基于改进YOLOX-s的机场跑道冰雪状态感知[J]. 上海交通大学学报, 2023, 57(10): 1292-1304.
[3]	李擎, 皇甫玉彬, 李江昀, 杨志方, 陈鹏, 王子涵. UConvTrans:全局和局部信息交互的双分支心脏图像分割[J]. 上海交通大学学报, 2023, 57(5): 570-581.
[4]	段继忠, 钱青青. 基于SIDWT和迭代自一致性的快速并行成像重建方法[J]. 上海交通大学学报, 2023, 57(5): 582-592.
[5]	黄鹤, 胡凯益, 李战一, 王会峰, 茹锋, 王珺. 融合MCAP和GRTV正则化的无人机航拍建筑物图像去雾方法[J]. 上海交通大学学报, 2023, 57(3): 366-378.
[6]	陈俊宇, 田凌. 基于船厂分段时空数据的分段状态识别及转运监测[J]. 上海交通大学学报, 2023, 57(1): 24-35.
[7]	韩明月, 王英林. 基于预训练语言模型的语法错误纠正方法[J]. 上海交通大学学报, 2022, 56(11): 1554-1560.
[8]	朱文敏, 骆晓萌, 范秀敏, 张磊, 蔡俊祺. 基于完整可达域分析的虚拟人作业姿态生成方法[J]. 上海交通大学学报, 2022, 56(10): 1409-1419.
[9]	程斌, 黄斌, 李得睿. 基于平行激光测距的图像自标定方法[J]. 上海交通大学学报, 2022, 56(7): 850-857.
[10]	闫青, 鲁建厦, 江伟光, 邵益平, 汤洪涛, 李英德. 考虑双端口布局的紧致化仓储系统堆垛机路径优化[J]. 上海交通大学学报, 2022, 56(7): 858-867.
[11]	郭涛, 刘明明, 曹蕾, 胡京招, 洪国军, 尤云祥. 疏浚泵内泥沙颗粒的瞬态追踪数值方法[J]. 上海交通大学学报, 2022, 56(5): 656-663.
[12]	李钰, 杨道勇, 刘玲亚, 王易因. 利用生成对抗网络实现水下图像增强[J]. 上海交通大学学报, 2022, 56(2): 134-142.
[13]	黄鹤, 吴琨, 李昕芮, 王珺, 王会峰, 茹锋. 自适应插值飞蛾扑火优化的多特征粒子滤波车辆跟踪算法[J]. 上海交通大学学报, 2022, 56(2): 143-155.
[14]	郭加伟, 许志杰, 何其昌. 虚拟装配手势库设计及人机工效评估优化[J]. 上海交通大学学报, 2022, 56(2): 127-133.
[15]	杨逸文, 贺越生, 王春香, 杨明. 一种面向固态激光雷达的简易标定与验证方法[J]. 上海交通大学学报, 2021, 55(S2): 98-102.