Track Research -> June 2014
Paper ID 1904
Title LogStore: A Workload-aware,Adaptable Key-Value Store on Hybrid Storage Systems
1. Q1: Overall Rating
( Required, Visible To Authors Only After Decision Notification )
Accept
Weak Accept
Neutral
Weak Reject
--Reject
2. Q2: Are there specific revisions that could raise your rating in the previous question?
( Optional )
Yes
--No
3. Q3: Summary of the paper (what is being proposed and in what context) and a brief justification of your overall recommendation. One paragraph
( Required, Visible To Authors During Feedback and After Decision Notification )
This paper introduces a hybrid (memory-SSD-HDD) key-value storage system, named LogStore. The authors integrate state-of-the-art techniques to build their key-value store, such as log-structured storage technique and SST. Unlike other existing hybrid key-value stores that use SSDs as a data cache, the authors build their system by using SSDs as a staging area and not a cache such that the data is either placed on SSDs or HDDs. They introduce cost model to estimate the read/write performance and propose optimizations on compactions. Some experiments are launched to compare it with LevelDB with or without SSD support.
4. Q4: Three (or more) strong points about the paper (Please be precise and explicit; clearly explain the value and nature of the contribution).
( Required, Visible To Authors During Feedback and After Decision Notification )
1. The paper did solid work on building a hybrid key-value store system.
2. Several compaction optimization techniques are proposed.
3. The review of modern key-value stores and their key techniques is well written.
5. Q5: Three (or more) weak points about the paper (Please indicate clearly whether the paper has any mistakes, missing related work, or results that cannot be considered a contribution; write it so that the authors can understand what are seen as negative aspect
( Required, Visible To Authors During Feedback and After Decision Notification )
1. The paper lacks of novel ideas.
2. The experiment section can be improved to better demonstrate the benefit of the proposed techniques.
3. The paper doesn't show superiority over other hybrid storage systems.
6. Q6: Relevant for PVLDB
( Required, Visible To Authors During Feedback and After Decision Notification )
--YES
NO
7. Q7: Novelty (Please give a high novelty ranking to papers on new topics, opening new fields, or proposing truly new ideas; assign medium ratings for delta papers and papers on well known topics but still with some valuable contribution).
( Required, Visible To Authors During Feedback and After Decision Notification )
Highly novel
Novel
--With some new ideas
Novelty unclear
Ideas are too simple (say how in Q5 or 12)
Ideas are not new (say why in Q5 or 12)
Same ideas published before (say where in Q12)
8. Q8: Significance
( Required, Visible To Authors During Feedback and After Decision Notification )
The paper is going to start a new line of research and products
--Improvement over existing work
No impact
9. Q9: Technical Depth and Quality of Content
( Required, Visible To Authors During Feedback and After Decision Notification )
Excellent work
Solid work
--Syntactically complete but with limited contribution
Insignificant contribution
Questionable work
10. Q10: Experiments
( Required, Visible To Authors During Feedback and After Decision Notification )
Very nicely support the claims made in the paper
--OK, but certain claims are not covered by the experiments
Obscure, not really sure what is going on and what the experiments show
Not applicable, there are no experiments
11. Q11: Presentation
( Required, Visible To Authors During Feedback and After Decision Notification )
Excellent: careful, logical, elegant, easy to understand
--Reasonable: improvements needed
Sub-standard: would require heavy rewrite
12. Q12: Detailed Evaluation (Contribution, Pros/Cons, Errors); please number each point
( Required, Visible To Authors During Feedback and After Decision Notification )
1. The paper builds a key-value hybrid storage system based on log-structured technique. Even though many optimizations are proposed to improve the compaction performance, I think the novelty of the paper is limited. The authors claimed four contributions in the paper. But unfortunately, many of them are not well demonstrated.
2. The first claimed contribution in the paper "An analytical cost model to estimate the performance of hybrid SSD/HDD storage setups". As you mentioned in the paper (S 4.4), the model suffers from limitations. The model does not consider the compaction cost, which I think has great impact on the performance. It is surely that considering compaction will make the model more complicated, but without considering compaction cost I have doubt about the accuracy. Moreover, I think the experimental result (S 6.1.3) fails to demonstrate the effectiveness of the cost model. How did you set s to draw the analytical bar in Fig.10? Is it 0.9? You use YCSB to generate Zipfian skew of s=0.9. Is it the reason why the result is so great? But in practical it is nontrivial to set s in your model. I think using Zipfian generator to evaluate Zipfian workload model (with the same parameter) is not a good idea. But if your system can adaptively learn the distribution paramter online, that should be OK.
3. The second claimed contribution. "A new statistics-driven compaction process that retains only the hottest data on the SSD." But the effect of the proposed compaction technique is not clear. How much benefit is gained from the proposed compaction method? If you can provide experimental results showing superiority over other compaction techniques, that would be better (it is better to add a separate subsection in S6 to show the effectiveness of statistics-driven compaction). I also suggest to compare with other statistics-driven compaction methods (I believe there should exist other relavant statistics-driven compaction techniques).
4. The migration between SSD and HDD relies on level histograms that track accesses to key ranges stored in every level. The advantages of level histograms are not explicitly shown. It is better to provide deep analysis and experimentally show its advantages.
5. The experiment section only compares LogStore to LevelDB-HDD and LevelDB-SSD. It is really interesting to see the comparison result between hybrid storage systems, e.g., comparing LogStore with [22].
13. Q13: If revision is required, list specific revisions you seek from the Authors
( Optional, Visible To Authors During Feedback and After Decision Notification )
14. Q14: Rate Your Own confidence in this review
( Optional )
Expert in this problem
--Knowledgeable in this sub-area
Generally aware of the area
Had to use common sense and general knowledge
15. Q15: Confidential Comments for the PC Chairs (Please add any information that may help us reach a decision)
( Optional )
16. Q16: Name and Affiliation of External Expert (!) Reviewer (if applicable)
( Optional )