Track Research -> June 2014
Paper ID 211
Title A Confidence-Aware Approach for Truth Discovery on Long-Tail Data
1. Q1: Overall Rating
( Required, Visible To Authors Only After Decision Notification )
Accept
--Weak Accept
Neutral
Weak Reject
Reject
2. Q2: Are there specific revisions that could raise your rating in the previous question?
( Optional )
--Yes
No
3. Q3: Summary of the paper (what is being proposed and in what context) and a brief justification of your overall recommendation. One paragraph
( Required, Visible To Authors During Feedback and After Decision Notification )
This paper focuses on the truth discovery problem from the inconsistent sources with the long-tail pattern, i.e., the number of claims made by sources follows the power law distribution. A statistics-based method is proposed to automatically detect truths, which can also score the confidence for each source.
Several real data sets are analyzed and tested to verify the correctness of the assumptions and effectiveness of the proposed model.
4. Q4: Three (or more) strong points about the paper (Please be precise and explicit; clearly explain the value and nature of the contribution).
( Required, Visible To Authors During Feedback and After Decision Notification )
1. A very significant and practical problem is studied.
2. The performance experiments are rather detailed measuring many factors.
3. The paper is well written and organized.
5. Q5: Three (or more) weak points about the paper (Please indicate clearly whether the paper has any mistakes, missing related work, or results that cannot be considered a contribution; write it so that the authors can understand what are seen as negative aspect
( Required, Visible To Authors During Feedback and After Decision Notification )
1. The novelty of the proposed statistics-based method is neutral.
2. The errors must follow the Gaussian distribution constraint.
3. It is not clear whether the proposed model can be suitable for more real-life applications.
6. Q6: Relevant for PVLDB
( Required, Visible To Authors During Feedback and After Decision Notification )
--YES
NO
7. Q7: Novelty (Please give a high novelty ranking to papers on new topics, opening new fields, or proposing truly new ideas; assign medium ratings for delta papers and papers on well known topics but still with some valuable contribution).
( Required, Visible To Authors During Feedback and After Decision Notification )
Highly novel
Novel
--With some new ideas
Novelty unclear
Ideas are too simple (say how in Q5 or 12)
Ideas are not new (say why in Q5 or 12)
Same ideas published before (say where in Q12)
8. Q8: Significance
( Required, Visible To Authors During Feedback and After Decision Notification )
The paper is going to start a new line of research and products
--Improvement over existing work
No impact
9. Q9: Technical Depth and Quality of Content
( Required, Visible To Authors During Feedback and After Decision Notification )
Excellent work
--Solid work
Syntactically complete but with limited contribution
Insignificant contribution
Questionable work
10. Q10: Experiments
( Required, Visible To Authors During Feedback and After Decision Notification )
Very nicely support the claims made in the paper
--OK, but certain claims are not covered by the experiments
Obscure, not really sure what is going on and what the experiments show
Not applicable, there are no experiments
11. Q11: Presentation
( Required, Visible To Authors During Feedback and After Decision Notification )
--Excellent: careful, logical, elegant, easy to understand
Reasonable: improvements needed
Sub-standard: would require heavy rewrite
12. Q12: Detailed Evaluation (Contribution, Pros/Cons, Errors); please number each point
( Required, Visible To Authors During Feedback and After Decision Notification )
1. The paper has two key constraints. The first one is that the claim number of different sources follows the power law distribution; The second one is that errors follow Gaussian distribution with the mean 0. If these two constraints hold, the proposed method is more like a Gaussian distribution or a chi-squared distribution inference. Of course, the proposed model is
simple but effective in this case.
2. The proposed three examples are similar to crowd sourcing applications. My concern is, for so many available crowd sourcing literatures which may contain the confidence evaluation of different sources, whether the same two constraints can be recognized for the scenarios or datasets they adopt.
3. Whether a broader spectrum of applications can follow these two constraints, such as database fusion in a company or sensor acquisition data?
4. For example1, my understanding is that volunteers usually tend to refer to some public materials before editing the population of different cities, instead of guessing or remembering the number. Correlations should exist between different users in this case.
5. For example3, what should be the mean like? If the mean is not 0, is it still reasonable to measure the source confidence based on error variance in whatever case?
6. For section 3.2.4, can you prove more iterations will surely lead to more accurate results? Have you tested the effect of iteration number on the errors?
13. Q13: If revision is required, list specific revisions you seek from the Authors
( Optional, Visible To Authors During Feedback and After Decision Notification )
See Q12 for details.
14. Q14: Rate Your Own confidence in this review
( Optional )
Expert in this problem
--Knowledgeable in this sub-area
Generally aware of the area
Had to use common sense and general knowledge
15. Q15: Confidential Comments for the PC Chairs (Please add any information that may help us reach a decision)
( Optional )
16. Q16: Name and Affiliation of External Expert (!) Reviewer (if applicable)
( Optional )