Results of the public multiformat listening test (July 2014)
These are the summary results of the multiformat listening test. The purpose was to test Opus, AAC and Ogg Vorbis at 96 kbps against a classic MP3 128 kbps, and find out which codec produces the best sound quality.
Summary
| ♛1 | Opus 1.1 with opus-tools-0.1.9-win32 |
opusenc --bitrate 96 in.wav out.opus | |
| ♛2 | AAC iTunes 11.2.2 via qaac 2.41 |
qaac --cvbr 96 -o out.mp4 in.wav | |
| ♛3 | Ogg Vorbis aoTuV Beta6.03 |
venc603 -q2.2 in.wav out.ogg | |
| ♛3 | MP3 LAME 3.99.5 |
lame -V5 in.wav out.mp3 *used 29% more bitrate. |
The sound quality of Opus, AAC, Ogg Vorbis and MP3 was evaluated by the public blind listening test. As a standard procedure of the ABC/HR methodology, the lossy audio files were paired with lossless original before the test. Thirty eight donators from public rated their perceived fidelity of those lossy files. Thirty five diverse music tracks and five speech tracks were used as test tracks.
The Opus encoder 1.1, the AAC encoder of iTunes, and the aoTuV at 96 kbps and the LAME at 128 kbps were tested. The most common response was on or between "Imperceptible" and "Perceptible, but not annoying".
The Opus codec was the clear winner, and the AAC encoded by Apple iTunes won the second place. The Ogg Vorbis was the third. The MP3, which was allowed to use 30kbps more bitrate than those contenders, was tied to Ogg Vorbis in joint third place.
FAAC (libfaac), which was known to be inferior before the test, was used as low quality anchors.
Contents
- How to interpret the charts
- Ratings of all tracks (40 tracks)
- All sets of tracks (5 sets, each 8 tracks)
- Ratings of sets of tracks
- Ratings of each track
- Donators, tracks and contributions (363 results, 339 valid results)
- Ratings of each donator
- Ratings of the top 12 donators
- Bitrates of tracks
- Bitrate vs ratings 2D scatter plot
- Listen and download tracks
- Post screening
- Bootstrap analysis
- ANOVA analysis
- Formatted ratings
- Download logs
- Contact
How to interpret the charts
Each chart is drawn with 6 codecs on the X axis and the rating given (1.0 to 5.0) on the Y axis. The mean rating given to each codec is indicated by the middle point of each vertical I-shaped line segment. Each I-shaped segment represents the 95% confidence interval of the mean rating (using bootstrap analysis) for each codec.
Ratings of all tracks (40 tracks)
All sets of tracks (5 sets, each 8 tracks)
Ratings of sets of tracks
| Set name | Opus 1.1 | AAC Apple | Ogg Vorbis | MP3 LAME | FAAC 96k | FAAC q30 | |
|---|---|---|---|---|---|---|---|
| Set name | Opus 1.1 | AAC Apple | Ogg Vorbis | MP3 LAME | FAAC 96k | FAAC q30 | |
%22%3E%3Ctext%20fill%3D%22rgb(0%2C51%2C255)%22%20stroke%3D%22none%22%20font-family%3D%22%26quot%3BCourier%20New%26quot%3B%2Cmonospace%22%20font-size%3D%2220px%22%20font-style%3D%22normal%22%20font-weight%3D%22bold%22%20text-decoration%3D%22normal%22%20x%3D%2211%22%20y%3D%2216%22%20text-anchor%3D%22middle%22%20dominant-baseline%3D%22alphabetic%22%3EC%3C%2Ftext%3E%3C%2Fg%3E%3C%2Fg%3E%3C%2Fsvg%3E){kind=small} | Chris's set of tracks | 4.66 | 4.26 | 4.22 | 4.22 | 2.62 | 1.15 |
%22%3E%3Ctext%20fill%3D%22rgb(0%2C187%2C0)%22%20stroke%3D%22none%22%20font-family%3D%22%26quot%3BCourier%20New%26quot%3B%2Cmonospace%22%20font-size%3D%2220px%22%20font-style%3D%22normal%22%20font-weight%3D%22bold%22%20text-decoration%3D%22normal%22%20x%3D%2211%22%20y%3D%2216%22%20text-anchor%3D%22middle%22%20dominant-baseline%3D%22alphabetic%22%3ET%3C%2Ftext%3E%3C%2Fg%3E%3C%2Fg%3E%3C%2Fsvg%3E){kind=small} | Tracks randomly picked from album libraries | 4.70 | 4.42 | 4.41 | 4.29 | 2.82 | 1.22 |
%22%3E%3Ctext%20fill%3D%22rgb(255%2C107%2C217)%22%20stroke%3D%22none%22%20font-family%3D%22%26quot%3BCourier%20New%26quot%3B%2Cmonospace%22%20font-size%3D%2220px%22%20font-style%3D%22normal%22%20font-weight%3D%22bold%22%20text-decoration%3D%22normal%22%20x%3D%2211%22%20y%3D%2216%22%20text-anchor%3D%22middle%22%20dominant-baseline%3D%22alphabetic%22%3EM%3C%2Ftext%3E%3C%2Fg%3E%3C%2Fg%3E%3C%2Fsvg%3E){kind=small} | Mpeg Sequences | 4.75 | 4.48 | 4.03 | 4.24 | 2.76 | 1.17 |
%22%3E%3Ctext%20fill%3D%22rgb(51%2C153%2C255)%22%20stroke%3D%22none%22%20font-family%3D%22%26quot%3BCourier%20New%26quot%3B%2Cmonospace%22%20font-size%3D%2220px%22%20font-style%3D%22normal%22%20font-weight%3D%22bold%22%20text-decoration%3D%22normal%22%20x%3D%2211%22%20y%3D%2216%22%20text-anchor%3D%22middle%22%20dominant-baseline%3D%22alphabetic%22%3ER%3C%2Ftext%3E%3C%2Fg%3E%3C%2Fg%3E%3C%2Fsvg%3E){kind=small} | Tracks picked from previous 2011 AAC test | 4.60 | 4.39 | 4.25 | 4.28 | 2.48 | 1.21 |
%22%3E%3Ctext%20fill%3D%22rgb(247%2C0%2C101)%22%20stroke%3D%22none%22%20font-family%3D%22%26quot%3BCourier%20New%26quot%3B%2Cmonospace%22%20font-size%3D%2220px%22%20font-style%3D%22normal%22%20font-weight%3D%22bold%22%20text-decoration%3D%22normal%22%20x%3D%2211%22%20y%3D%2216%22%20text-anchor%3D%22middle%22%20dominant-baseline%3D%22alphabetic%22%3EA%3C%2Ftext%3E%3C%2Fg%3E%3C%2Fg%3E%3C%2Fsvg%3E){kind=small} | Tracks picked from new HA thread 103989 | 4.56 | 4.45 | 4.28 | 4.16 | 2.58 | 1.23 |
Ratings of each track
Chris's set of tracks
SinceAlways
Waiting
velvet
trumpet
girl
Can't Wait Until Tonight (Dry Wurlitzer Mix)
35_SQAM_glockenspiel_cut
Robots_old
Tracks randomly picked from album libraries
Asleep__4.11-4.31_
Greatest_Love_of_All_2min57
Hey Tonight
Severance__1.31-1.51_
Shinsho_pool_3min45_4min4
SlavesOfFear
The Chastising of Renegade
TrosYGareg
Mpeg Sequences
4-Sound-English-male.441
9-Have-big-expensive-car.441
12-German-male-speech.441
15-Good-evening.441
21-classic.441
24-Greensleeves-Korean-male-speech.441
25-This-is-the-end.441
27-last-song-drums-and-trampets.441
Tracks picked from previous 2011 AAC test
bonhemian_rhapsody
clapton_44k
Coral
ExitMusic
liberate
NewYorkCity
sandman
take_your_finger_from_my_hair
Tracks picked from new HA thread 103989
Changes
Girl_In_The_Fire__Sample_
Hotel California
Jupiter, the Bringer of Jollity
Last_Of_The_Mohicanz__Sample_
Only Time
Through The Fire And Flames
With Love (Outro)
Donators, tracks and contributions (363 results, 339 valid results)
Ratings of each donator
Ratings of the top 12 donators
Ratings submitted by kiavan92 (38 results)
Ratings submitted by Alexxander (35 results)
Ratings submitted by anonymous2 (35 results)
Ratings submitted by win32vb (32 results)
Ratings submitted by Kamedo2 (25 results)
Ratings submitted by Steve Forte Rio (24 results)
Ratings submitted by IgorC (21 results)
Ratings submitted by Caspian (16 results)
Ratings submitted by Gainless (13 results)
Ratings submitted by Joungmin Lee (11 results)
Ratings submitted by 2Bdecided (10 results)
Ratings submitted by singlethread (10 results)
Bitrates of tracks
In each codec, settings were calibrated to provide ~96kbps on a large variety of music on average, except for the MP3 and anchors.
These are the file size based bitrates.
Sample Length Opus AAC Ogg MP3 FAAC 96 FAAC q30 -------------------------------------------------------------------------- 1 18.191 82 103 97 144 98 58 SinceAlways 2 20.155 111 106 107 137 98 54 Waiting 3 11.879 112 114 124 169 98 55 velvet 4 10.031 133 101 101 156 98 60 trumpet 5 15.957 122 101 134 155 98 51 girl 6 29.933 116 99 111 144 97 54 Can't Wait Until Tonight (Dry Wurlitzer Mix) 7 23.45 131 85 79 77 83 41 35_SQAM_glockenspiel_cut 8 30 100 108 105 128 98 44 Robots_old 9 20 103 109 116 148 98 48 Asleep__4.11-4.31_ 10 20.178 103 100 89 120 98 54 Greatest_Love_of_All_2min57 11 20 92 102 96 131 98 55 Hey Tonight 12 20 120 100 94 131 98 54 Severance__1.31-1.51_ 13 18.924 104 109 88 108 98 48 Shinsho_pool_3min45_4min4 14 22 90 100 107 131 98 56 SlavesOfFear 15 20 104 109 122 146 98 49 The Chastising of Renegade 16 22.304 124 99 96 127 98 55 TrosYGareg 17 9.028 96 95 99 121 98 46 4-Sound-English-male.441 18 9.707 119 104 113 154 99 48 9-Have-big-expensive-car.441 19 7.84 100 95 104 116 98 47 12-German-male-speech.441 20 8.566 94 99 100 129 98 52 15-Good-evening.441 21 9.656 112 105 89 133 98 52 21-classic.441 22 8.72 133 101 121 176 99 45 24-Greensleeves-Korean-male-speech.441 23 9.184 98 99 95 125 98 53 25-This-is-the-end.441 24 9.966 104 105 112 128 99 48 27-last-song-drums-and-trampets.441 25 28.505 101 106 94 120 98 53 bonhemian_rhapsody 26 14.986 104 107 122 145 98 53 clapton_44k 27 12.701 104 107 102 135 98 48 Coral 28 20.106 111 101 85 141 98 47 ExitMusic 29 5.007 97 116 143 160 98 54 liberate 30 29.849 93 103 102 131 98 55 NewYorkCity 31 30.002 97 99 106 134 98 49 sandman 32 22.092 114 109 133 138 98 60 take_your_finger_frin_my_head 33 29.932 113 106 119 137 98 51 Changes 34 13.742 121 111 108 126 98 53 Girl_In_The_Fire__Sample_ 35 30.762 106 110 112 143 98 52 Hotel California 36 29.234 105 103 89 123 98 56 Jupiter, the Bringer of Jollity 37 25.481 104 110 114 147 98 48 Last_Of_The_Mohicanz__Sample_ 38 24.671 96 110 100 130 98 48 Only Time 39 29.789 97 99 117 135 98 59 Through The Fire And Flames 40 16.403 103 103 99 144 98 56 With Love (Outro) -------------------------------------------------------------------------- Mean 18.973 106.7 103.7 106.1 135.5 97.6 51.7 Unit second kbps kbps kbps kbps kbps kbps Sample Length Opus AAC Ogg MP3 FAAC 96 FAAC q30
Bitrate vs ratings 2D scatter plot
Listen and download tracks
The Links on the left side are the lossless original tracks, and the right side are the lossy tracks.
Post screening
Invalid results were discarded according to the following criteria, which had been made public prior to the test:
- If a listener ranked the low anchor at 5.0 on a sample, the listener's results for that sample were discarded.
- If a listener ranked the mid-low anchor at 5.0 on a sample, the listener's results for that sample were discarded.
- If a listener ranked the low anchor higher than the mid-low anchor on a sample, the listener's results for that sample were discarded.
- If a listener ranked the reference worse than 4.5 on a sample, the listener's results for that sample were discarded.
- If a listener ranked the reference worse than 5.0 on 25% or more of submitted results, all of that listener's results were discarded.
- If a listener submitted 25% or more invalid results, all of the listener's results were discarded.
The discarded results were less than 7% of all results.
Bootstrap analysis
bootstrap.py --compare-all -p 1000000 -s 1000000 aq23.txt
bootstrap.py v1.0 2011-02-03
Copyright (C) 2011 Gian-Carlo Pascutto
License Affero GPL version 3 or later <http://www.gnu.org/licenses/agpl.html>
Reading from: C:\Users\PCC\Downloads\friedman124\aq23.txt
Read 6 treatments, 40 samples => 15 comparisons
Means:
AppleAAC Opus Ogg MP3V5 FAAC96k FAACq30
4.400 4.653 4.235 4.237 2.652 1.194
Unadjusted p-values:
Opus Ogg MP3V5 FAAC96k FAACq30
AppleAAC 0.000* 0.023* 0.016* 0.000* 0.000*
Opus - 0.000* 0.000* 0.000* 0.000*
Ogg - - 0.981 0.000* 0.000*
MP3V5 - - - 0.000* 0.000*
FAAC96k - - - - 0.000*
Opus is better than AppleAAC (p=0.000)
Ogg is worse than AppleAAC (p=0.023)
Ogg is worse than Opus (p=0.000)
MP3V5 is worse than AppleAAC (p=0.016)
MP3V5 is worse than Opus (p=0.000)
FAAC96k is worse than AppleAAC (p=0.000)
FAAC96k is worse than Opus (p=0.000)
FAAC96k is worse than Ogg (p=0.000)
FAAC96k is worse than MP3V5 (p=0.000)
FAACq30 is worse than AppleAAC (p=0.000)
FAACq30 is worse than Opus (p=0.000)
FAACq30 is worse than Ogg (p=0.000)
FAACq30 is worse than MP3V5 (p=0.000)
FAACq30 is worse than FAAC96k (p=0.000)
p-values adjusted for multiple comparison:
Opus Ogg MP3V5 FAAC96k FAACq30
AppleAAC 0.000* 0.044* 0.043* 0.000* 0.000*
Opus - 0.000* 0.000* 0.000* 0.000*
Ogg - - 0.981 0.000* 0.000*
MP3V5 - - - 0.000* 0.000*
FAAC96k - - - - 0.000*
Opus is better than AppleAAC (p=0.000)
Ogg is worse than AppleAAC (p=0.044)
Ogg is worse than Opus (p=0.000)
MP3V5 is worse than AppleAAC (p=0.043)
MP3V5 is worse than Opus (p=0.000)
FAAC96k is worse than AppleAAC (p=0.000)
FAAC96k is worse than Opus (p=0.000)
FAAC96k is worse than Ogg (p=0.000)
FAAC96k is worse than MP3V5 (p=0.000)
FAACq30 is worse than AppleAAC (p=0.000)
FAACq30 is worse than Opus (p=0.000)
FAACq30 is worse than Ogg (p=0.000)
FAACq30 is worse than MP3V5 (p=0.000)
FAACq30 is worse than FAAC96k (p=0.000)
ANOVA analysis
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis
Number of listeners: 40
Critical significance: 0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings
Source of Degrees Sum of Mean
variation of Freedom squares Square F p
Total 239 385.20
Testers (blocks) 39 4.52
Codecs eval'd 5 369.44 73.89 1282.39 0.00E+000
Error 195 11.24 0.06
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.106
Means:
Opus AppleAAC MP3V5 Ogg FAAC96k FAACq30
4.65 4.40 4.24 4.24 2.65 1.19
---------------------------- p-value Matrix ---------------------------
AppleAAC MP3V5 Ogg FAAC96k FAACq30
Opus 0.000* 0.000* 0.000* 0.000* 0.000*
AppleAAC 0.003* 0.002* 0.000* 0.000*
MP3V5 0.976 0.000* 0.000*
Ogg 0.000* 0.000*
FAAC96k 0.000*
-----------------------------------------------------------------------
Opus is better than AppleAAC, MP3V5, Ogg, FAAC96k, FAACq30
AppleAAC is better than MP3V5, Ogg, FAAC96k, FAACq30
MP3V5 is better than FAAC96k, FAACq30
Ogg is better than FAAC96k, FAACq30
FAAC96k is better than FAACq30
Formatted ratings
The ratings parsed for ANOVA analysis, Bootstrap analysis and graphmaker 5.
Opus AAC Ogg MP3 FAAC96k FAACq30 %feature 10 0.1.9-win32 iTunes 11, qaac 2.41 aoTuV b6.03 LAME 3.99.5 faac-1.28-mod from rarewares faac-1.28-mod from rarewares %feature 11 --bitrate 96 --cvbr 96 -q2.2 -V5 -b 96 -q 30 %feature 12 107kbps 104kbps 106kbps 136kbps 98kbps 52kbps %genre Chris's set of tracks 4.960 4.860 4.660 4.420 2.630 1.330 4.580 4.490 4.030 4.290 2.610 1.070 4.345 4.382 4.373 4.373 2.464 1.127 4.840 4.270 4.240 3.910 2.440 1.040 4.492 3.238 3.954 4.077 2.577 1.238 4.433 3.767 3.378 4.578 2.567 1.056 4.700 4.558 4.425 4.492 3.067 1.242 4.911 4.533 4.667 3.611 2.633 1.056 %genre Tracks randomly picked from album libraries 4.885 4.908 4.785 4.885 2.946 1.062 4.667 4.367 4.322 4.267 2.967 1.100 4.400 4.567 4.133 4.056 2.811 1.200 4.650 3.950 4.000 4.283 2.717 1.367 4.811 4.544 4.289 3.878 2.922 1.400 4.913 4.313 4.850 4.738 2.850 1.225 4.760 4.430 4.710 4.270 2.880 1.180 4.488 4.250 4.150 3.963 2.488 1.238 %genre Mpeg Sequences 4.970 4.820 3.770 4.470 3.010 1.220 4.711 4.044 4.278 4.167 2.589 1.122 4.820 4.800 3.590 4.090 2.710 1.190 4.314 4.514 4.243 4.186 2.486 1.186 4.686 4.571 3.957 3.943 2.886 1.429 5.000 4.317 4.233 4.400 2.917 1.000 4.888 4.413 4.200 4.563 2.700 1.225 4.571 4.343 3.971 4.057 2.771 1.000 %genre Tracks picked from previous 2011 AAC test 4.629 4.529 4.043 3.714 2.486 1.257 4.286 4.214 4.457 4.300 2.414 1.043 4.940 4.500 4.530 4.620 2.570 1.290 4.933 4.433 4.133 3.800 2.333 1.000 4.243 4.429 4.614 4.500 2.486 1.486 4.800 4.067 4.133 4.483 2.383 1.000 4.925 4.325 4.400 4.725 2.663 1.363 4.043 4.657 3.700 4.129 2.529 1.214 %genre Tracks picked from new HA thread 103989 4.363 4.213 4.425 4.250 2.538 1.188 4.363 4.500 4.125 4.025 2.938 1.100 4.814 4.586 4.129 4.457 2.729 1.229 4.483 4.417 3.800 4.100 2.733 1.300 4.678 4.211 4.667 4.322 2.267 1.178 4.514 4.686 4.200 4.186 2.529 1.114 4.650 4.475 4.675 3.913 2.375 1.513 4.650 4.500 4.175 3.988 2.488 1.175 %samples SinceAlways Guitar %samples Waiting Pops %samples velvet Perc. %samples trumpet Wind %samples girl Techno %samples Can't Wait Until Tonight (Dry Wurlitzer Mix) Pops %samples 35_SQAM_glockenspiel_cut Inst. %samples Robots_old Techno %samples Asleep__4.11-4.31_ Strings %samples Greatest_Love_of_All_2min57 Pops %samples Hey Tonight Rock %samples Severance__1.31-1.51_ Pops %samples Shinsho_pool_3min45_4min4 Pops %samples SlavesOfFear Metal %samples The Chastising of Renegade Rock %samples TrosYGareg Pops %samples 4-Sound-English-male.441 Voice %samples 9-Have-big-expensive-car.441 Pops %samples 12-German-male-speech.441 Voice %samples 15-Good-evening.441 Voice %samples 21-classic.441 Classic %samples 24-Greensleeves-Korean-male-speech.441 Voice %samples 25-This-is-the-end.441 Voice %samples 27-last-song-drums-and-trampets.441 Perc. %samples bonhemian_rhapsody Rock %samples clapton_44k Guitar %samples Coral Pops %samples ExitMusic Vocal %samples liberate Metal %samples NewYorkCity Pops %samples sandman Rock %samples take_your_finger_from_my_hair Rock %samples Changes Rock %samples Girl_In_The_Fire__Sample_ Rock %samples Hotel California Rock %samples Jupiter, the Bringer of Jollity Classic %samples Last_Of_The_Mohicanz__Sample_ Electro %samples Only Time Newage %samples Through The Fire And Flames Metal %samples With Love (Outro) Techno %link_path tracks/ .wav .opus1.1.b96k.opus .qaac.cvbr96k.mp4 .aotuv.q2.2.ogg .lame3.99.5.v5.mp3 .faac.abr96k.mp4 .faac.vbr30q.mp4
Download logs
You can download a ZIP file containing all logs. Encryption keys and parsed text included.
Contact
Kamedo2: Twitter@kamedo2