Video Quality Evaluation Tool Experimental Documentation
The experimental tools use two tools from the full reference:
- MSU VQMT
- NetFlix vmaf
Because psnr is commonly used as a video quality evaluation index, this experiment is mainly for testing and comparison, so only psnr is used as the index of the two tools
the difference:
-
VQMT is a commercial software, the current free version can support HD, FullHD, 4k, 8k resolution and Support for 200+ video & image formats
Download link: MSU Video Quality Measurement Tool (VMAF, PSNR, VQM, SSIM, NIQE, etc) (compression.ru)
- The main differences between Free and Pro are as follows: (the pro version is expensive, 1 device 999$, you can choose ProDemo, which is also a free version, but it officially says that it does not guarantee the correct data)
Features & Components Free ProDemo Pro PREMIUM GPU metrics − + + + HDR − + + + Windows Console − + + + Linux Console − + + + MAC (iOS) support − Planned Planned Planned Python Interface − + + + Unlimited installations + + − + Automatical activation N/A N/A − + Free updates + + 1 year 1 year Correct results (not DEMO) + − + + Usage in companies − + + + -
vmaf only supports yuv and y4m formats, and also provides several parameters to choose from (psnr, psnr_hvs, ssim, ms-ssim, ciede)
Download link: Releases Netflix/vmaf (github.com)
Operation method:
-
VQMT has a graphical interface to directly open video files for comparison
It also has a variety of parameters that can be used. The optional parameters are as follows: (You can see that there is Netflix VMAF here)
输入原视频及参考视频URL或者打开文件后便可以开始进行分析，在新版本中还会自动保存分析的结果，只是不能进行批处理，需要Pro版本的命令行才能支持批处理工作。
-
vmaf does not have a graphical interface, you need to use command line tools to operate
Command line usage guide vmaf/README.md at master Netflix/vmaf (github.com)
-
instruction:
vmaf -r E:\common\tang3.yuv -d E:\common\tang4.yuv -w 960 -h 960 -p 420 -b 8 -o output.log -
Related parameters:
Reference .y4m or .yuv (-r/--reference) is required Usage: vmaf [options] Supported options: --reference/-r $path: path to reference .y4m or .yuv --distorted/-d $path: path to distorted .y4m or .yuv --width/-w $unsigned: width --height/-h $unsigned: height --pixel_format/-p: $string pixel format (420/422/444) --bitdepth/-b $unsigned: bitdepth (8/10/12/16) --model/-m $params: model parameters, colon ":" delimited `path=` path to model file `version=` built-in model version `name=` name used in log (optional) --output/-o $path: output file --xml: write output file as XML (default) --json: write output file as JSON --csv: write output file as CSV --sub: write output file as subtitle --threads $unsigned: number of threads to use --feature $string: additional feature Additional Metrics 其他的可选参数 psnr, psnr_hvs, ssim, ms-ssim, ciede A number of addtional metrics are supported. Enable these metrics with the --feature flag. # psnr, psnr_hvs, ssim, ms-ssim, ciede --feature psnr \ --feature psnr_hvs \ --feature float_ssim \ --feature float_ms_ssim --feature ciede --feature cambi --cpumask: $bitmask restrict permitted CPU instruction sets --frame_cnt $unsigned: maximum number of frames to process --subsample: $unsigned compute scores only every N frames --quiet/-q: disable FPS meter when run in a TTY --no_prediction/-n: no prediction, extract features only --version/-v: print version and exit
-
output:
-
VQMT
- Y psnr (signal-to-noise ratio Peak signal-to-noise ratio);
Convert the picture to YCbCr format, and then only calculate the PSNR of the Y component, which is the brightness component. The typical peak signal-to-noise ratio value in image compression is between 30 and 40dB, the higher the better.
-
Output report:
Metric: PSNR Color component: Y Files: constrait20fps.mp4 compared to: native_video.mp4 # native_video.mp4 0 48.089218 1 48.097214 2 40.957611 3 48.148323 4 47.362122 ...... ...... 10837 18.317478 10838 18.341671 10839 18.339325 10840 18.337685 10841 18.108231 10842 18.247126 10843 18.247503 10844 18.245304 total psnr 34.472847 mean 47.133518 harmonic mean 46.136684 min. val 16.455748 max. val 61.386471 min. frame 3623 max. frame 389 std dev 4.385028 variance 19.228470
-
vmaf
- No features added:
<VMAF version="2.3.1"> <params qualityWidth="960" qualityHeight="960" /> <fyi fps="20.94" /> <frames> <frame frameNum="0" integer_adm2="0.978963" integer_adm_scale0="0.998125" integer_adm_scale1="0.995882" integer_adm_scale2="0.990478" integer_adm_scale3="0.956561" integer_motion2="0.000000" integer_motion="0.000000" integer_vif_scale0="0.549783" integer_vif_scale1="0.619952" integer_vif_scale2="0.648450" integer_vif_scale3="0.675513" vmaf="59.246770" /> <frame frameNum="1" integer_adm2="0.991671" integer_adm_scale0="0.998279" integer_adm_scale1="0.994878" integer_adm_scale2="0.994040" integer_adm_scale3="0.986643" integer_motion2="0.852844" integer_motion="6.023301" integer_vif_scale0="0.815954" integer_vif_scale1="0.879384" integer_vif_scale2="0.898762" integer_vif_scale3="0.913492" vmaf="85.293165" /> ...... ...... </frames> <pooled_metrics> <metric name="integer_adm2" min="0.413357" max="0.999776" mean="0.928834" harmonic_mean="0.923530" /> <metric name="integer_adm_scale0" min="0.720963" max="1.001498" mean="0.978396" harmonic_mean="0.977443" /> <metric name="integer_adm_scale1" min="0.340115" max="1.003616" mean="0.940059" harmonic_mean="0.933984" /> <metric name="integer_adm_scale2" min="0.277522" max="0.999417" mean="0.926124" harmonic_mean="0.916255" /> <metric name="integer_adm_scale3" min="0.325116" max="0.999697" mean="0.911883" harmonic_mean="0.906875" /> <metric name="integer_motion2" min="0.000000" max="35.799564" mean="6.326431" harmonic_mean="3.413652" /> <metric name="integer_motion" min="0.000000" max="67.485893" mean="7.641753" harmonic_mean="3.884414" /> <metric name="integer_vif_scale0" min="0.096378" max="0.978172" mean="0.592144" harmonic_mean="0.571733" /> <metric name="integer_vif_scale1" min="0.105987" max="0.990578" mean="0.660542" harmonic_mean="0.639917" /> <metric name="integer_vif_scale2" min="0.104521" max="0.992963" mean="0.687723" harmonic_mean="0.666811" /> <metric name="integer_vif_scale3" min="0.098240" max="0.995127" mean="0.710630" harmonic_mean="0.689127" /> <metric name="vmaf" min="0.000000" max="96.823158" mean="63.415244" harmonic_mean="38.980743" /> </pooled_metrics> <aggregate_metrics /> </VMAF>- Add psnr:
<VMAF version="e1d466c"> <params qualityWidth="576" qualityHeight="324" /> <fyi fps="41.98" /> <frames> <frame frameNum="0" integer_adm2="0.962084" integer_adm_scale0="0.946338" integer_adm_scale1="0.939006" integer_adm_scale2="0.957474" integer_adm_scale3="0.980887" integer_motion2="0.000000" integer_motion="0.000000" integer_vif_scale0="0.505712" integer_vif_scale1="0.879061" integer_vif_scale2="0.937873" integer_vif_scale3="0.964301" psnr_y="34.760779" psnr_cb="39.229987" psnr_cr="41.349703" vmaf="83.856285" /> ... </frames> <pooled_metrics> <metric name="integer_adm2" min="0.921008" max="0.962084" mean="0.934506" harmonic_mean="0.934463" /> ... <metric name="psnr_y" min="29.640688" max="34.760779" mean="30.755064" harmonic_mean="30.727905" /> <metric name="psnr_cb" min="38.019979" max="39.229987" mean="38.449441" harmonic_mean="38.447866" /> <metric name="psnr_cr" min="40.649266" max="41.353846" mean="40.991910" harmonic_mean="40.991083" /> <metric name="vmaf" min="71.176557" max="87.181420" mean="76.668905" harmonic_mean="76.510006" /> </pooled_metrics> <aggregate_metrics />
in conclusion:
In terms of use, VQMT is simpler than vmaf because it has a graphical interface. In terms of functions, VQMT brought me two surprises. The first point is that it can use vmaf metric, which is equivalent to covering the functions of vmaf. The other point is that it is planned in the future. Stream support and more parameters are provided in the version of , but I don't know if it can be used in the free version.
Stream support
Allows VQMT read and estimate the quality of streamed video
For video evaluation with original video and reference video, the free version can meet the requirements after testing, and the new free version 14.1 of VQMT can also support high resolution and multiple pixel formats.
From a comprehensive evaluation, vmaf can use the command line to perform batch processing operations, which is one of its advantages, but VQMT is easy to use and powerful, and for the current usage requirements, it does not need to perform batch processing operations, so it is relatively VQMT is currently the best choice for full-reference video quality evaluation.