miun.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Statistical quality of experience analysis: on planning the sample size and statistical significance testing
RISE AB Acreo. (Realistic3D)ORCID iD: 0000-0001-5060-9402
Deggendorf Institute of Technology (DIT), University of Applied Sciences, Deggendorf.
2018 (English)In: Journal of Electronic Imaging (JEI), ISSN 1017-9909, E-ISSN 1560-229X, Vol. 27, no 5, p. 053013-1-053013-11, article id 053013Article in journal (Refereed) Published
Abstract [en]

This paper analyzes how an experimenter can balance errors in subjective video quality tests betweenthe statistical power of finding an effect if it is there and not claiming that an effect is there if the effect is not there,i.e., balancing Type I and Type II errors. The risk of committing Type I errors increases with the number ofcomparisons that are performed in statistical tests. We will show that when controlling for this and at thesame time keeping the power of the experiment at a reasonably high level, it is unlikely that the number oftest subjects that are normally used and recommended by the International Telecommunication Union (ITU),i.e., 15 is sufficient but the number used by the Video Quality Experts Group (VQEG), i.e., 24 is more likelyto be sufficient. Examples will also be given for the influence of Type I error on the statistical significance ofcomparing objective metrics by correlation. We also present a comparison between parametric and nonparametricstatistics. The comparison targets the question whether we would reach different conclusions on the statisticaldifference between the video quality ratings of different video clips in a subjective test, based on thecomparison between the student T-test and the Mann–Whitney U-test. We found that there was hardly a differencewhen few comparisons are compensated for, i.e., then almost the same conclusions are reached. Whenthe number of comparisons is increased, then larger and larger differences between the two methods arerevealed. In these cases, the parametric T-test gives clearly more significant cases, than the nonparametrictest, which makes it more important to investigate whether the assumptions are met for performing a certaintest.

Place, publisher, year, edition, pages
IS&T - the Society for Imaging Science and Technology , 2018. Vol. 27, no 5, p. 053013-1-053013-11, article id 053013
Keywords [en]
Type-I error, video quality, statistical significance, quality of experience, Student T-test, Bonferroni, Mann–Whitney U-test, parametric versus nonparametric test
National Category
Electrical Engineering, Electronic Engineering, Information Engineering Media Engineering Signal Processing Communication Systems
Identifiers
URN: urn:nbn:se:miun:diva-34508DOI: 10.1117/1.JEI.27.5.053013ISI: 000449229800015Scopus ID: 2-s2.0-85054069504OAI: oai:DiVA.org:miun-34508DiVA, id: diva2:1251082
Funder
Knowledge Foundation, 20160194
Note

Copyright (2018) Society of Photo-Optical Instrumentation Engineers. One print or electronic copy may be made for personal use only. Systematic reproduction and distribution, duplication of any material in this paper for a fee or for commercial purposes, or modification of the content of the paper are prohibited.

Available from: 2018-09-26 Created: 2018-09-26 Last updated: 2018-12-06Bibliographically approved

Open Access in DiVA

fulltext(1742 kB)292 downloads
File information
File name FULLTEXT01.pdfFile size 1742 kBChecksum SHA-512
2047984830a72ea9c89ec856e15f55080d966c8867df51d3271480cb47de5ce50d5fe39370e4e97f26b331d9b9fc5ada3cdd80fdef625d3a5d398c761744c026
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopus

Authority records BETA

Brunnström, Kjell

Search in DiVA

By author/editor
Brunnström, Kjell
In the same journal
Journal of Electronic Imaging (JEI)
Electrical Engineering, Electronic Engineering, Information EngineeringMedia EngineeringSignal ProcessingCommunication Systems

Search outside of DiVA

GoogleGoogle Scholar
Total: 292 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 336 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf