Home > Archive > Compression > January 2007 > JPEG-1/2000 HDPhoto measurements online
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
JPEG-1/2000 HDPhoto measurements online
|
|
| Thomas Richter 2007-01-09, 3:56 am |
| Hi folks,
a new benchmark for JPEG-1/JPEG2000 and Microsoft HD Photo has been
recently uploaded to http://www.math.tu-berlin.de/~thor/img/
You'll find there the following:
o) One directory per test picture. The pictures are taken from
the ITU test set, some have been used by the JPEG internally. Sorry,
I don't have the web space to host the pictures themselves. If you
care or need the images for visual inspection, I can sent them to you
either electronically, or on DVD. The compressed test image set (all
images compressed to all qualities) for visual reconstruction
takes on two DVDs now. If you want other/more images included, please
let me know.
Each directory contains a PDF file showing the PSRN over bpp plot,
the same data as ASCII table, one common table, and one per codec.
HDPhoto and JPEG-1 unfortunately do not have a rate-allocation, so
the table entries are not easily related to each other. The plots
are potentially more meaningful.
Take the measurements with the usual "grain of salt" as they only
depict PSNR, which relates to visual quality only to some degree.
You should know that.
The plots cover:
- Traditional JPEG-1 baseline (red)
- ITU-T 851 (green)
- JPEG-1 with QM coder (yellow)
All of the above use the IJG codec.
- JPEG 2000 (blue) using the Pegasus codec.
- HDPhoto (pink) using the Microsoft HDPhoto kit.
o) In the same (top) directory, the file "jpegcompare", which
is a bash script that generated the images.
o) In the same (top) directory, hdphoto.patch, which is a patch
file that must be applied to Microsoft's HDPhoto kit to make
it compilable, at least on GNU g++ on an AMD64 platform.
The source is unfortunately not portable (unlike what the kit says) to
other compilers and architectures without fixing a couple of
"Microsoftisms" (#pragma once, characters must be unsigned, and no, an
"int" is not enough to hold a pointer on my machine, and disabling all
warnings is almost always a bad idea). I put the patch there to enable
you to reproduce the test, to let you know that I'm not messing with the
code to fake the test, and potentially, to have MS fix their code. Tsk,
tsk...
--
Some results (brief review):
- On all tests I've made, JPEG2000 performs best PSNR wise.
I could not find an image where HDPhoto would outperform it, but
maybe it requires still some tweaking on the command line. Any hints
what to do would be appreciated. You have the source of the script -
I'm just doing the baseline coding here and nothing fancy.
- On all the tests I've made, HDPhoto outperforms JPEG-1 and its
derivatives, i.e. ITU-T 851.
- ITU-T 851 usually performs as good as JPEG-1 with QM coder,
but the code seems to fail on some images, i.e. there's likely
still a bug in the IJG implementation I have used.
Visually, HDPhoto is a compromize between wavelets and DCT.
This compromize gives "blurred DCT artefacts" as far as I can
tell. They appear "less blocky" than classical DCT artefacts
and go half-way into the direction of the "only blurry" wavelet
artefacts. In my thinking, it's a matter of taste what you
dislike the least. (-:
So long,
Thomas
| |
| mihai cartoaje 2007-01-10, 3:56 am |
| Thomas Richter wrote:
> Hi folks,
>
> a new benchmark for JPEG-1/JPEG2000 and Microsoft HD Photo has been
> recently uploaded to http://www.math.tu-berlin.de/~thor/img/
>
> You'll find there the following:
>
> o) One directory per test picture. The pictures are taken from
> the ITU test set, some have been used by the JPEG internally. Sorry,
> I don't have the web space to host the pictures themselves. If you
> care or need the images for visual inspection, I can sent them to you
> either electronically, or on DVD. The compressed test image set (all
> images compressed to all qualities) for visual reconstruction
> takes on two DVDs now. If you want other/more images included, please
> let me know.
>
> Each directory contains a PDF file showing the PSRN over bpp plot,
> the same data as ASCII table, one common table, and one per codec.
> HDPhoto and JPEG-1 unfortunately do not have a rate-allocation, so
> the table entries are not easily related to each other. The plots
> are potentially more meaningful.
>
> Take the measurements with the usual "grain of salt" as they only
> depict PSNR, which relates to visual quality only to some degree.
> You should know that.
>
> The plots cover:
>
> - Traditional JPEG-1 baseline (red)
> - ITU-T 851 (green)
> - JPEG-1 with QM coder (yellow)
>
> All of the above use the IJG codec.
>
> - JPEG 2000 (blue) using the Pegasus codec.
> - HDPhoto (pink) using the Microsoft HDPhoto kit.
>
> o) In the same (top) directory, the file "jpegcompare", which
> is a bash script that generated the images.
>
> o) In the same (top) directory, hdphoto.patch, which is a patch
> file that must be applied to Microsoft's HDPhoto kit to make
> it compilable, at least on GNU g++ on an AMD64 platform.
> The source is unfortunately not portable (unlike what the kit says) to
> other compilers and architectures without fixing a couple of
> "Microsoftisms" (#pragma once, characters must be unsigned, and no, an
> "int" is not enough to hold a pointer on my machine, and disabling all
> warnings is almost always a bad idea). I put the patch there to enable
> you to reproduce the test, to let you know that I'm not messing with the
> code to fake the test, and potentially, to have MS fix their code. Tsk,
> tsk...
>
> --
>
> Some results (brief review):
>
> - On all tests I've made, JPEG2000 performs best PSNR wise.
> I could not find an image where HDPhoto would outperform it, but
> maybe it requires still some tweaking on the command line. Any hints
> what to do would be appreciated. You have the source of the script -
> I'm just doing the baseline coding here and nothing fancy.
>
> - On all the tests I've made, HDPhoto outperforms JPEG-1 and its
> derivatives, i.e. ITU-T 851.
>
> - ITU-T 851 usually performs as good as JPEG-1 with QM coder,
> but the code seems to fail on some images, i.e. there's likely
> still a bug in the IJG implementation I have used.
>
> Visually, HDPhoto is a compromize between wavelets and DCT.
> This compromize gives "blurred DCT artefacts" as far as I can
> tell. They appear "less blocky" than classical DCT artefacts
> and go half-way into the direction of the "only blurry" wavelet
> artefacts. In my thinking, it's a matter of taste what you
> dislike the least. (-:
>
> So long,
> Thomas
I compared with the spiht version of my codec for lena gray:
JPEG2000 1.31655 39.5003
HD Photo 1.73333 40.4054
libima (float) 1.31653 40.3181
libima (int) 1.31653 39.6623
libima (float) 1.73330 42.2382
libima (int) 1.73330 41.7094
The source code for libima is at
http://geocities.com/repstsb/libima.html
The spiht version can be selected by adding -DSPIHT to the variable
CFLAGS in the makefile. On the website is also the image I used.
| |
| Sachin Garg 2007-01-10, 6:56 pm |
|
On Jan 9, 2:39 pm, Thomas Richter <t...@math.TU-Berlin.DE> wrote:
> Hi folks,
>
> a new benchmark for JPEG-1/JPEG2000 and Microsoft HD Photo has been
> recently uploaded tohttp://www.math.tu-berlin.de/~thor/img/
>
> You'll find there the following:
>
> o) One directory per test picture. The pictures are taken from
> the ITU test set, some have been used by the JPEG internally. Sorry,
> I don't have the web space to host the pictures themselves. If you
> care or need the images for visual inspection, I can sent them to you
> either electronically, or on DVD. The compressed test image set (all
> images compressed to all qualities) for visual reconstruction
> takes on two DVDs now. If you want other/more images included, please
> let me know.
>
> Each directory contains a PDF file showing the PSRN over bpp plot,
> the same data as ASCII table, one common table, and one per codec.
> HDPhoto and JPEG-1 unfortunately do not have a rate-allocation, so
> the table entries are not easily related to each other. The plots
> are potentially more meaningful.
>
> Take the measurements with the usual "grain of salt" as they only
> depict PSNR, which relates to visual quality only to some degree.
> You should know that.
>
> The plots cover:
>
> - Traditional JPEG-1 baseline (red)
> - ITU-T 851 (green)
> - JPEG-1 with QM coder (yellow)
>
> All of the above use the IJG codec.
>
> - JPEG 2000 (blue) using the Pegasus codec.
> - HDPhoto (pink) using the Microsoft HDPhoto kit.
>
> o) In the same (top) directory, the file "jpegcompare", which
> is a bash script that generated the images.
>
> o) In the same (top) directory, hdphoto.patch, which is a patch
> file that must be applied to Microsoft's HDPhoto kit to make
> it compilable, at least on GNU g++ on an AMD64 platform.
> The source is unfortunately not portable (unlike what the kit says) to
> other compilers and architectures without fixing a couple of
> "Microsoftisms" (#pragma once, characters must be unsigned, and no, an
> "int" is not enough to hold a pointer on my machine, and disabling all
> warnings is almost always a bad idea). I put the patch there to enable
> you to reproduce the test, to let you know that I'm not messing with the
> code to fake the test, and potentially, to have MS fix their code. Tsk,
> tsk...
>
> --
>
> Some results (brief review):
>
> - On all tests I've made, JPEG2000 performs best PSNR wise.
> I could not find an image where HDPhoto would outperform it, but
> maybe it requires still some tweaking on the command line. Any hints
> what to do would be appreciated. You have the source of the script -
> I'm just doing the baseline coding here and nothing fancy.
>
> - On all the tests I've made, HDPhoto outperforms JPEG-1 and its
> derivatives, i.e. ITU-T 851.
>
> - ITU-T 851 usually performs as good as JPEG-1 with QM coder,
> but the code seems to fail on some images, i.e. there's likely
> still a bug in the IJG implementation I have used.
>
> Visually, HDPhoto is a compromize between wavelets and DCT.
> This compromize gives "blurred DCT artefacts" as far as I can
> tell. They appear "less blocky" than classical DCT artefacts
> and go half-way into the direction of the "only blurry" wavelet
> artefacts. In my thinking, it's a matter of taste what you
> dislike the least. (-:
This helps confirm the results reported by MSU folks in a similar
benchmark in October last year.
But Bill Crow, Microsoft's program manager for HD Photo, raised a few
questions regarding the methodology used and some justification on why
HDPhoto might still be better. Unfortunately, all those concerns still
apply to your results too.
Basically, it was regarding use of PSNR as a quality metric and more
importantly, use of these OLD low quality images for testing codecs
which are to be used on today's high quality/resolution images.
More details at http://www.c10n.info/archives/454
Another question I had is regarding relative speed of two algorithms.
AFAIK, primary reason for jpeg2000 not getting too popular in
mainstream is its slow speed compared to jpeg (it ofcourse still has
its niche applications which can afford to pay the price of extra cpu
cycles), and one of the HD Photo's selling point is its fast speed. But
none of the benchmark's from MS or others has measured this, yet.
Sachin Garg [India]
www.sachingarg.com | www.c10n.info
| |
| cr88192 2007-01-10, 6:56 pm |
|
"Sachin Garg" <schngrg@gmail.com> wrote in message
news:1168443484.269476.234600@i39g2000hsf.googlegroups.com...
>
> On Jan 9, 2:39 pm, Thomas Richter <t...@math.TU-Berlin.DE> wrote:
<snip>
[color=darkred]
>
> This helps confirm the results reported by MSU folks in a similar
> benchmark in October last year.
>
> But Bill Crow, Microsoft's program manager for HD Photo, raised a few
> questions regarding the methodology used and some justification on why
> HDPhoto might still be better. Unfortunately, all those concerns still
> apply to your results too.
>
> Basically, it was regarding use of PSNR as a quality metric and more
> importantly, use of these OLD low quality images for testing codecs
> which are to be used on today's high quality/resolution images.
>
> More details at http://www.c10n.info/archives/454
>
>
> Another question I had is regarding relative speed of two algorithms.
> AFAIK, primary reason for jpeg2000 not getting too popular in
> mainstream is its slow speed compared to jpeg (it ofcourse still has
> its niche applications which can afford to pay the price of extra cpu
> cycles), and one of the HD Photo's selling point is its fast speed. But
> none of the benchmark's from MS or others has measured this, yet.
>
yes, I wonder this myself.
though I have limited understanding of the DWT, given its general design, I
am uncertain speed-wise how it compares with, for example, the DCT and MDCT
transforms.
actually, I still wonder why apparently about no one afaik has used the MDCT
for image transforms, though my initial results are looking promising (on
the quality and compression front).
I have now considered optimizing the transform (converting it to an
integer-based version and factoring), seeing how fast I can get the thing
(ie: if it is possible to approach the speed of the good ol' DCT).
then again, this effort is likely to amount to nothing anyways, but oh
well... (alternating between this and working on my VM more...).
> Sachin Garg [India]
> www.sachingarg.com | www.c10n.info
>
| |
| Thomas Richter 2007-01-10, 6:56 pm |
| Sachin Garg wrote:
>
> This helps confirm the results reported by MSU folks in a similar
> benchmark in October last year.
>
> But Bill Crow, Microsoft's program manager for HD Photo, raised a few
> questions regarding the methodology used and some justification on why
> HDPhoto might still be better. Unfortunately, all those concerns still
> apply to your results too.
>
> Basically, it was regarding use of PSNR as a quality metric and more
> importantly, use of these OLD low quality images for testing codecs
> which are to be used on today's high quality/resolution images.
Well, I agree that PSNR is pretty bad, which however poses the question
what is better? If MS is proposing some specific quality metric, and I
can find an implementation of it, I'd be glad to run the tests with this
metric again. I already tested with SSIM, but found really no
significant difference in the results. I haven't tested multi-scale SIM.
Concerning the test images, I've also included a couple of larger,
higher resolution images, not just the ITU-T images (you'll see the
image dimensions in the text files in each directory). If you have any
suggestions concerning additional test images, please go along.
> More details at http://www.c10n.info/archives/454
>
>
> Another question I had is regarding relative speed of two algorithms.
> AFAIK, primary reason for jpeg2000 not getting too popular in
> mainstream is its slow speed compared to jpeg (it ofcourse still has
> its niche applications which can afford to pay the price of extra cpu
> cycles), and one of the HD Photo's selling point is its fast speed. But
> none of the benchmark's from MS or others has measured this, yet.
I really haven't tried to measure speed yet. I think it definitely
depends on the coding options you choose (and your favorite compiler and
its options, too... :-)
If anyone cares, I could try to setup a speed measurement as well, but
as stated, your results may differ significantly, and really depend on
the codec you use. I have no doubt that JPEG-1 will be faster, though. I
don't know where HDPhoto would place itself, so maybe this is at least
some kind of a question worth to consider.
Thanks for the link, BTW.
So long,
Thomas
| |
| Thomas Richter 2007-01-10, 6:56 pm |
| cr88192 wrote:
>
> yes, I wonder this myself.
Partly because it is an ill-posed question, to some degree. It's not a
standard that is faster than another, but an implementation.
> though I have limited understanding of the DWT, given its general design, I
> am uncertain speed-wise how it compares with, for example, the DCT and MDCT
> transforms.
Microsoft's HDPhoto is basically a MDCT, i.e. overlapped blocking DCT
lookalike.
> actually, I still wonder why apparently about no one afaik has used the MDCT
> for image transforms, though my initial results are looking promising (on
> the quality and compression front).
Shrug. I played a little bit with it and found that it didn't go along
very well with quantization if you use a "naively" implemented MDCT, so
probably you've done something smarter than I did.
So long,
Thomas
| |
| Thomas Richter 2007-01-10, 6:56 pm |
| mihai cartoaje wrote:
> I compared with the spiht version of my codec for lena gray:
>
> JPEG2000 1.31655 39.5003
> HD Photo 1.73333 40.4054
> libima (float) 1.31653 40.3181
> libima (int) 1.31653 39.6623
> libima (float) 1.73330 42.2382
> libima (int) 1.73330 41.7094
>
> The source code for libima is at
> http://geocities.com/repstsb/libima.html
> The spiht version can be selected by adding -DSPIHT to the variable
> CFLAGS in the makefile. On the website is also the image I used.
Sorry, but please first fix your code.
thor@cutter:~/src/libima> make
Makefile:71: depend: Datei oder Verzeichnis nicht gefunden
echo '# Module dependencies' >>depend
gcc -O2 -s -mcpu=i586 -fpic -Wstrict-prototypes -DFP_TYPE=float
-DINT_TYPE=long -DFLOATING -MM bitor.c bito.c encode.c encode_haar.c
encode_int.c codec.c transformsh.c crc.c encode_d.c biti.c decode.c
decode_haar.c decode_int.c codec.c transformsh.c crc.c decode_d.c png.c
png2ima.c ima2png.c compare.c >>depend
`-mcpu=' is deprecated. Use `-mtune=' or '-march=' instead.
bitor.c:0: error: CPU you selected does not support x86-64 instruction set
make: *** [depend] Fehler 1
This is easily fixed, but:
thor@cutter:~/src/libima> png2ima shipbig.png -c 0.5
the image is 1280 x 1024 pixels with 3 channels 8 bits
just sits there and loops forever. I haven't attempted to debug your
code, but I would expect that you need to take of architectures where
longs and pointers are longer than 32 bytes.
So long,
Thomas
| |
| cr88192 2007-01-10, 6:56 pm |
|
"Thomas Richter" <thor@math.tu-berlin.de> wrote in message
news:50kveaF1fkml1U2@mid.dfncis.de...
> cr88192 wrote:
>
>
> Partly because it is an ill-posed question, to some degree. It's not a
> standard that is faster than another, but an implementation.
>
makes sense.
at first I had figured that DCT would always be slower than LPC, but then I
figured, a sufficiently involved LPC coder risks being slower than DCT.
thus, LPC only really has a strong advantage when the transform is kept
fairly simple.
could be similar with wavelets, but then again, I am not all that fammiliar
with them.
>
> Microsoft's HDPhoto is basically a MDCT, i.e. overlapped blocking DCT
> lookalike.
>
ok.
I read a spec for hdphoto at one point, but didn't see where they defined
the transform, so wasn't sure what exactly they were doing...
>
> Shrug. I played a little bit with it and found that it didn't go along
> very well with quantization if you use a "naively" implemented MDCT, so
> probably you've done something smarter than I did.
>
dunno what you mean by "naively" here.
my MDCT is basically just a plain MDCT and a windowing function, though
currently using floats (as of yet I haven't implemented/tested an integer
version).
I am not sure how you were doing quantizing.
given what I had seen for dumped quantizer tables, the "shape" seems a
little different than for the normal DCT (a preference for most harshly
quantizing the middle-frequency coefficients).
as for quantizing, it could be the algo I am using to generate the quantizer
tables. the basic algo was originally developed in my case for experimenting
with MDCT-based audio-compression (since I had little idea what I was doing
at the time, I originally assumed it to be about like a 1D version of image
compression, and vice versa).
the algo also seemed to work well on graphics using DCT (actually, a lot
better than it did on audio), so I used it there.
for my MDCT codec, I just reused the same basic algo, but tweaked some
things slightly, mostly because the MDCT seems to generate larger values and
thus the existing quantization value limit (1..255) just wasn't doing a very
good job.
I seem to be getting decent results even with fairly harsh quantization.
as noted, I am compressing an 800x600 image down to about 11kB.
the file takes about 75kB at 85% and 100kB at 95% quality as per gimp.
11kB in my format tests (a hard-coded 20% quality) gives similar results to
a 20kB jpeg, which is to say, hardly looking all that great, but not the
pixelated mess of the 11kB jpeg.
of course, it does matter to me that encoding/decoding be fast, and I have
yet to optimize the MDCT transform...
for all I know, the improvement in compression could be due to "something
else" besides the MDCT (for example, how I am approaching VLC-coding the
coefficients, or that internally I drop the dynamic range for the YUV planes
in order to help avoid artifacting, ...).
so, yeah, there is still a lot of uncertainty...
> So long,
> Thomas
| |
| Sachin Garg 2007-01-10, 9:56 pm |
|
On Jan 11, 1:06 am, Thomas Richter <t...@math.tu-berlin.de> wrote:
> Sachin Garg wrote:
>
>
>
>
> Well, I agree that PSNR is pretty bad, which however poses the question
> what is better? If MS is proposing some specific quality metric, and I
> can find an implementation of it, I'd be glad to run the tests with this
> metric again. I already tested with SSIM, but found really no
> significant difference in the results. I haven't tested multi-scale SIM.
As you might have read, he just mentioned better in subjective visual
analysis :-)
> Concerning the test images, I've also included a couple of larger,
> higher resolution images, not just the ITU-T images (you'll see the
> image dimensions in the text files in each directory). If you have any
> suggestions concerning additional test images, please go along.
None, but its still something to be considered.
Moreover, as most algorithms are tuned for the kind of data which is
used for testing when developing the algorithms, it wouldn't be
surprising if Jpeg2000's results are skewed on these popular images and
if Microsoft's results are skewed on their own test set ;-)
>
>
> I really haven't tried to measure speed yet. I think it definitely
> depends on the coding options you choose
Yes ofcourse, but thats not much different from choice of parameters
for compression ratio testing.
> (and your favorite compiler and
> its options, too... :-)
For this part, maybe the default build system with which the code comes
can be used.
> If anyone cares, I could try to setup a speed measurement as well, but
> as stated, your results may differ significantly, and really depend on
> the codec you use. I have no doubt that JPEG-1 will be faster, though. I
> don't know where HDPhoto would place itself, so maybe this is at least
> some kind of a question worth to consider.
In one of my old tests (but that was just for lossless compression),
both jpeg2000 (kakadu) and png were around same speed, both 3 times
slower than jpeg-ls. Am ofcourse curious to know where HDPhoto would
stand, but didn't quite got to actually trying it out.
> Thanks for the link, BTW.
You're welcome.
Sachin Garg [India]
www.sachingarg.com | www.c10n.info
| |
| mihai cartoaje 2007-01-11, 6:56 pm |
| mihai cartoaje wrote:
> Thomas Richter wrote:
>
> I compared with the spiht version of my codec for lena gray:
>
> JPEG2000 1.31655 39.5003
> HD Photo 1.73333 40.4054
> libima (float) 1.31653 40.3181
> libima (int) 1.31653 39.6623
> libima (float) 1.73330 42.2382
> libima (int) 1.73330 41.7094
>
> The source code for libima is at
> http://geocities.com/repstsb/libima.html
> The spiht version can be selected by adding -DSPIHT to the variable
> CFLAGS in the makefile. On the website is also the image I used.
I was using a different version of the image. I retested using the
version from the university of waterloo website:
libima (float) 1.31653 40.3275
libima (int) 1.31653 39.6576
libima (float) 1.73331 42.2463
libima (int) 1.73331 41.7153
| |
| Thomas Richter 2007-01-14, 9:55 pm |
| Sachin Garg wrote:
>
> As you might have read, he just mentioned better in subjective visual
> analysis :-)
Well, "never thrust a statistic you haven't faked yourself." (-;
Subjective quality means approximately nothing.
>
> None, but its still something to be considered.
I retested with the Kodak test image set. I would believe that the
images in there are too small to be useful nowadays, but anyhow. The
result isn't different at all.
> Moreover, as most algorithms are tuned for the kind of data which is
> used for testing when developing the algorithms, it wouldn't be
> surprising if Jpeg2000's results are skewed on these popular images and
> if Microsoft's results are skewed on their own test set ;-)
Shrug. Maybe. I don't think I've selected any test image "on purpose",
and I would be happy to test again with the MS test image set.
>
> Yes ofcourse, but thats not much different from choice of parameters
> for compression ratio testing.
>
>
> For this part, maybe the default build system with which the code comes
> can be used.
Well, just to give you an example: Our JPEG2000 code can take advantage
of multi-core CPUs, so it runs in parallel on two cores. The HDPhoto and
the IJG codec does not. Now, enable or disable this option? One can
argue that for the customer, the net result of speed he gets on his
machine is important, and if the codec is designed in a way that allows
it to take advantage of this, it should be done. On the other hand, it
can also be argued that this is unfair because neither IJG or MS do
this. You can also sacrifice a tiny little bit of quality for a huge
speed improvement by making some "ad hoc" assumptions on how typical
images look like. Is this good, is this bad?
Well, I would believe that whatever the tests will look like, someone
will complain, and you can certainly tune them in whatever direction you
want.
So long,
Thomas
|
|
|
|
|