Compression - ANNOUNCEMENT: New image format

Author

ANNOUNCEMENT: New image format

John Costella

2007-11-22, 6:56 pm

12:30 pm CST, November 22, 2007

I am pleased to announce the release of a new image format, which I
have somewhat mischievously dubbed "JPEG-Clear", which is optimized
for the following uses, among others:

* Transmission of images or image sequences compressed losslessly or
lossily through a limited bandwidth channel (such as the Internet),
where rapid, smooth, progressive, and antialiased rendering of the
image being downloaded is desirable (such as on a web page).

* Lossless compression of images or image sequences, particularly
where it is advantageous to have high-quality antialiased
thumbnails available rapidly, without any need for them to be
separately generated or stored.

* Lossy compression of images, where the ability to rapidly render
thumbnails or render to lower resolution is advantageous (e.g.
photo CDs).

* Transmission of video through a limited bandwidth channel where it
is desirable to dynamically subscribe to and unsubscribe from
additional detail according to the available bandwidth, without need
for replication of any information at the server.

The full details are posted at:

http://assassinationscience.com/johncostella/jpegclear

Benefits:
--------

* Recursive "top-down" algorithm works equally well for images and
image sequences of all sizes and resolutions.

* Progressive, high-quality, antialiased decoding of any image, where
the image simply gets "sharper" as more of it is downloaded.

* High-quality, antialiased thumbnail images at all powers of two are
already intrinsically contained within every image file or frame,
and can be rapidly extracted without decompressing the entire image.

* Both lossless and lossy compression, with a seamless transition
between the two, using a single algorithm.

* Lossless compression ratios comparable to those of PNG.

* Graceful handling of high compression ratios in lossy mode, with any
image able to be compressed to as little as 24 bytes.

* Images can be beautifully and rapidly enlarged by any power of two
using the "magic kernel" used by the library.

* Only integer arithmetic additions and subtractions performed; no
integer multiplication or divisions, or floating point operations.

* Supports all compilers and architectures, and executes identically
on
all systems.

* Uses only very simple mathematical processes and algorithms that
have
been in the public domain for decades.

* Time to decompress full image comparable to that of JPEG-2000.

* Calling applications can specify:
- lossless compression;
- auto-adaptive lossy compression within a "byte budget" that is
guaranteed not to be exceeded; or
- the compression parameters for each channel at each scale.

Negatives:
---------

* Adaptive lossy compression is up to ten times slower than lossless
compression or fixed lossy compression (it actually performs ten
times as many passes).

* Compression and full decompression times expected to be slower
than JPEG-XR (not tested).

* Lossy compression quality satisfactory for transmission and
display purposes, but not competitive with JPEG-2000 or JPEG-XR
for master image storage.

Supports:
--------

* Grayscale and color, in either the RGB or YCbCr colorspace.

* Alpha transparency channel.

* 8 bits per channel optimized, and up to 16 bits per channel
optimized,
simultaneously in the one application.

* 64-bit image sizes (each dimension up to 32 bits).

* Image data in essentially any regular image storage memory layout.

* Animated image sequences and video.

* Up to 4 GB of metadata, either supplied as bytes (such as for XML)
compressed by the library if possible, or as incompressible bits.

* Video can be separated efficiently into multiple resolution and/or
multiple compression-rate datastreams.

* "Container" format allows the rapid, high-quality, progressive
download of all images and animations on a web page at the same time
through only a single connection to the web server.

Requirements:
------------

* Any 32-bit or 64-bit ANSI C compiler on any architecture.

* Agreement to the "MIT" license, which allows you to do anything at
all with the library as long as you don't blame me if it doesn't
work.

Non-requirements:
----------------

* No other libraries required whatsoever.

* No licensing requirements or restrictions.

* No royalties sought or accepted.

* No patents known.

* No carbohydrates.

* Can be used in any open-source project.

* Can be used in any proprietary product.

* No requirement to divulge your source code.

* No requirement to divulge your existence or use of the library.

Supplied:
--------

* Sample console applications that convert between BMP and JPEG-Clear,
and between JPEG and JPEG-Clear.

* All source code, including the sample applications.

* Extensive documentation of the API and the source code itself.

* Windows GUI Progressive Download Simulator (with genuine use of the
library throughout; the only "simulation" is the throttling of the
data rate).

* Images on website illustrating the intrinsic antialiased thumbnails.

* Windows GUI application comparing lossy compression performance
of vanilla versions of JPEG, JPEG-2000, and JPEG-Clear.

Not yet supplied:
-----------------

* Code and documentation for the animation/video and web container
formats will be posted in the coming w

s.

Please ask here or contact me directly if you have any questions.

John Costella

========================================
=====================
Dr. John P. Costella
BE(Elec)(Hons), BSc(Hons), PhD(Physics), GradDipEd
9 Summerfield Drive, Mornington, Victoria 3931, Australia
+61 3 5973 6929, costella@bigpond.com, jpcostella@hotmail.com
http://assassinationscience.com/johncostella
========================================
=====================

Phil Carmody

2007-11-22, 6:56 pm

John Costella <jpcostella@hotmail.com> writes:
> * Lossless compression ratios comparable to those of PNG.

Don't just say they're comparable - show us the data.

Phil
--
Dear aunt, let's set so double the killer delete select all.
-- Microsoft voice recognition live demonstration

Industrial One

2007-11-22, 9:56 pm

On Nov 22, 11:30 am, John Costella <jpcoste...@hotmail.com> wrote:
> 12:30 pm CST, November 22, 2007
>
> I am pleased to announce the release of a new image format, which I
> have somewhat mischievously dubbed "JPEG-Clear", which is optimized
> for the following uses, among others:
>
> * Transmission of images or image sequences compressed losslessly or
> lossily through a limited bandwidth channel (such as the Internet),
> where rapid, smooth, progressive, and antialiased rendering of the
> image being downloaded is desirable (such as on a web page).
>
> * Lossless compression of images or image sequences, particularly
> where it is advantageous to have high-quality antialiased
> thumbnails available rapidly, without any need for them to be
> separately generated or stored.
>
> * Lossy compression of images, where the ability to rapidly render
> thumbnails or render to lower resolution is advantageous (e.g.
> photo CDs).
>
> * Transmission of video through a limited bandwidth channel where it
> is desirable to dynamically subscribe to and unsubscribe from
> additional detail according to the available bandwidth, without need
> for replication of any information at the server.
>
> The full details are posted at:
>
> http://assassinationscience.com/johncostella/jpegclear
>
> Benefits:
> --------
>
> * Recursive "top-down" algorithm works equally well for images and
> image sequences of all sizes and resolutions.
>
> * Progressive, high-quality, antialiased decoding of any image, where
> the image simply gets "sharper" as more of it is downloaded.
>
> * High-quality, antialiased thumbnail images at all powers of two are
> already intrinsically contained within every image file or frame,
> and can be rapidly extracted without decompressing the entire image.
>
> * Both lossless and lossy compression, with a seamless transition
> between the two, using a single algorithm.
>
> * Lossless compression ratios comparable to those of PNG.
>
> * Graceful handling of high compression ratios in lossy mode, with any
> image able to be compressed to as little as 24 bytes.
>
> * Images can be beautifully and rapidly enlarged by any power of two
> using the "magic kernel" used by the library.
>
> * Only integer arithmetic additions and subtractions performed; no
> integer multiplication or divisions, or floating point operations.
>
> * Supports all compilers and architectures, and executes identically
> on
> all systems.
>
> * Uses only very simple mathematical processes and algorithms that
> have
> been in the public domain for decades.
>
> * Time to decompress full image comparable to that of JPEG-2000.
>
> * Calling applications can specify:
> - lossless compression;
> - auto-adaptive lossy compression within a "byte budget" that is
> guaranteed not to be exceeded; or
> - the compression parameters for each channel at each scale.
>
> Negatives:
> ---------
>
> * Adaptive lossy compression is up to ten times slower than lossless
> compression or fixed lossy compression (it actually performs ten
> times as many passes).
>
> * Compression and full decompression times expected to be slower
> than JPEG-XR (not tested).
>
> * Lossy compression quality satisfactory for transmission and
> display purposes, but not competitive with JPEG-2000 or JPEG-XR
> for master image storage.
>
> Supports:
> --------
>
> * Grayscale and color, in either the RGB or YCbCr colorspace.
>
> * Alpha transparency channel.
>
> * 8 bits per channel optimized, and up to 16 bits per channel
> optimized,
> simultaneously in the one application.
>
> * 64-bit image sizes (each dimension up to 32 bits).
>
> * Image data in essentially any regular image storage memory layout.
>
> * Animated image sequences and video.
>
> * Up to 4 GB of metadata, either supplied as bytes (such as for XML)
> compressed by the library if possible, or as incompressible bits.
>
> * Video can be separated efficiently into multiple resolution and/or
> multiple compression-rate datastreams.
>
> * "Container" format allows the rapid, high-quality, progressive
> download of all images and animations on a web page at the same time
> through only a single connection to the web server.
>
> Requirements:
> ------------
>
> * Any 32-bit or 64-bit ANSI C compiler on any architecture.
>
> * Agreement to the "MIT" license, which allows you to do anything at
> all with the library as long as you don't blame me if it doesn't
> work.
>
> Non-requirements:
> ----------------
>
> * No other libraries required whatsoever.
>
> * No licensing requirements or restrictions.
>
> * No royalties sought or accepted.
>
> * No patents known.
>
> * No carbohydrates.
>
> * Can be used in any open-source project.
>
> * Can be used in any proprietary product.
>
> * No requirement to divulge your source code.
>
> * No requirement to divulge your existence or use of the library.
>
> Supplied:
> --------
>
> * Sample console applications that convert between BMP and JPEG-Clear,
> and between JPEG and JPEG-Clear.
>
> * All source code, including the sample applications.
>
> * Extensive documentation of the API and the source code itself.
>
> * Windows GUI Progressive Download Simulator (with genuine use of the
> library throughout; the only "simulation" is the throttling of the
> data rate).
>
> * Images on website illustrating the intrinsic antialiased thumbnails.
>
> * Windows GUI application comparing lossy compression performance
> of vanilla versions of JPEG, JPEG-2000, and JPEG-Clear.
>
> Not yet supplied:
> -----------------
>
> * Code and documentation for the animation/video and web container
> formats will be posted in the coming w

s.
>
> Please ask here or contact me directly if you have any questions.
>
> John Costella
>
> ========================================
=====================
> Dr. John P. Costella
> BE(Elec)(Hons), BSc(Hons), PhD(Physics), GradDipEd
> 9 Summerfield Drive, Mornington, Victoria 3931, Australia
> +61 3 5973 6929, coste...@bigpond.com, jpcoste...@hotmail.comhttp://assassinationscience.com/johncostella
> ========================================
=====================

Awesome! Now... if I were you, when I'd announce a new technology I
wouldn't yap my ass off about minor features nobody cares to know but
state good reasons why it should replace previous JPEG by comparing
sizes, quality, giving us a summary what JPEGClear has that JPEG or
JPEG2000 doesn't. Follow?

If the lossless engine is comparable to PNG then why the hell would we
need your shit?

He's Late

2007-11-23, 3:56 am

On Thu, 22 Nov 2007 19:27:48 -0800 (PST), Industrial One
<industrial_one@hotmail.com> wrote:

>Awesome! Now... if I were you, when I'd announce a new technology I
>wouldn't yap my ass off about minor features nobody cares to know but
>state good reasons why it should replace previous JPEG by comparing
>sizes, quality, giving us a summary what JPEGClear has that JPEG or
>JPEG2000 doesn't. Follow?
>
>If the lossless engine is comparable to PNG then why the hell would we
>need your shit?

Doesn't matter, he's way too late. The new HD Photo 16-bit-support image format
is already being supported in programs like PhotoLine 32. It's already doing all
that stuff. Started by Vista, one of the few things that Gates did right.

Industrial One

2007-11-23, 3:56 am

On Nov 22, 10:18 pm, He's Late <betterl...@never.net> wrote:

> Doesn't matter, he's way too late. The new HD Photo 16-bit-support image format
> is already being supported in programs like PhotoLine 32. It's already doing all
> that stuff. Started by Vista, one of the few things that Gates did right.

Who are you? And Vista = r

ing garbage masked by a sweet-smelling
perfume.

Thomas Richter

2007-11-23, 3:56 am

John Costella schrieb:
> 12:30 pm CST, November 22, 2007
>
>
> I am pleased to announce the release of a new image format, which I
> have somewhat mischievously dubbed "JPEG-Clear", which is optimized
> for the following uses, among others:
>
> * Transmission of images or image sequences compressed losslessly or
> lossily through a limited bandwidth channel (such as the Internet),
> where rapid, smooth, progressive, and antialiased rendering of the
> image being downloaded is desirable (such as on a web page).
>
> * Lossless compression of images or image sequences, particularly
> where it is advantageous to have high-quality antialiased
> thumbnails available rapidly, without any need for them to be
> separately generated or stored.
>
> * Lossy compression of images, where the ability to rapidly render
> thumbnails or render to lower resolution is advantageous (e.g.
> photo CDs).
>
> * Transmission of video through a limited bandwidth channel where it
> is desirable to dynamically subscribe to and unsubscribe from
> additional detail according to the available bandwidth, without need
> for replication of any information at the server.
>
>
> The full details are posted at:
>
> http://assassinationscience.com/johncostella/jpegclear

First of all, I wouldn't call it JPEG unless it is coming from the JPEG.
Strictly speaking, we don't have a trademark AFAIK, but still, I think
it is confusing and I would recommend not to do this. I don't think we
will come after you, but please, *don't do that*. Thanks!

Second point is that I think you should definitely measure compression
performance and publish test results on that. By performance I do not
only mean PSNR, but other metrics as well: PQS, M-SSIM, VDP. If you need
help on that, let me know.

Third, as you may or may not know, JPEG currently has an open call for
advanced image coding (AIC) where, if competitive, you may file your
format for discussion in the JPEG and finally can standardize it. I
highly recommend, though, to make measurements upfront.

> * Lossless compression ratios comparable to those of PNG.

You may want to compare to state of the art, which is for example
JPEG-LS. PNG is not a very reasonable lossless compressor.

> Negatives:
> ---------
>
> * Adaptive lossy compression is up to ten times slower than lossless
> compression or fixed lossy compression (it actually performs ten
> times as many passes).

Have you considered implementing a smarter rate allocation, either
a posteriori as in the EBCOT in JPEG2000, or an a priori algorithm that
estimates the rate from the coefficient statistics? If it needs "passes"
to find the rate, it looks like you can do better.

> * Compression and full decompression times expected to be slower
> than JPEG-XR (not tested).

As long as it is within the limits of JPEG2000, I wouldn't care as long
as it is reasonably better than JPEG-XR.

> * Lossy compression quality satisfactory for transmission and
> display purposes, but not competitive with JPEG-2000 or JPEG-XR
> for master image storage.

....which is too bad.

> * Grayscale and color, in either the RGB or YCbCr colorspace.

Please think about ICC profiles to support arbitrary color spaces.
(Besides, there is no "the RGB" color space)

So long,
Thomas

Miles Bader

2007-11-23, 3:56 am

Industrial One <industrial_one@hotmail.com> writes:
> On Nov 22, 10:18 pm, He's Late <betterl...@never.net> wrote:

Microsoft has a verrrry bad record of foisting really crappy data
formats on the world.

Previously they tried to foist a not-very-good HDR image format called
"scRGB". Here's a detail description of HDR encodings that discusses
some of the problems with scRGB:

http://www.anyhere.com/gward/hdrenc/hdr_encodings.html

"HD photo" seems to incorporate scRGB in some form, which doesn't bode
well for its suitability as an HDR encoding...

[Microsoft, of course, doesn't really care, as long as they own the
patents to everything, they're happy whether or not the result sucks.]
[color=darkred]
>
> Who are you? And Vista = r

ing garbage masked by a sweet-smelling
> perfume.

Well at least that's one things most people can agree on: Vista sucks.

-Miles

--
In New York, most people don't have cars, so if you want to kill a person, you
have to take the subway to their house. And sometimes on the way, the train
is delayed and you get impatient, so you have to kill someone on the subway.
[George Carlin]

Industrial One

2007-11-23, 3:56 am

On Nov 23, 12:43 am, Thomas Richter <t...@math.tu-berlin.de> wrote:
ncostella/jpegclear
>
> Strictly speaking, we don't have a trademark
> I don't think we will come after you
>
> So long,
> Thomas

What's this "we" shit, bro?

Thomas Richter

2007-11-23, 3:56 am

Industrial One schrieb:
> On Nov 23, 12:43 am, Thomas Richter <t...@math.tu-berlin.de> wrote:
> ncostella/jpegclear
>
> What's this "we" shit, bro?

"We" as in "the JPEG". Yes, I'm a JPEG member. (To be precise, of the
"SC29/WG1" committee of the ISO, whose slang name happens to be "JPEG").

So long,
Thomas

Thomas Richter

2007-11-23, 3:56 am

John Costella schrieb:
> 12:30 pm CST, November 22, 2007
>
>
> I am pleased to announce the release of a new image format, which I
> have somewhat mischievously dubbed "JPEG-Clear", which is optimized
> for the following uses, among others:

Update to that:

1) Please add a Makefile. Here's one:

CFLAGS = -Wall -pedantic -O3
CC = gcc
LDFLAGS = -lm

all: bmp_to_jpc jpc_to_bmp

bmp_to_jpc: bmp_to_jpc.o costella_base.o costella_bitstream.o
costella_bmp.o costella_eimage.o costella_eimage_convert.o
costella_eimage_esimage.o costella_eimage_esimage_convert.o
costella_esimage.o costella_esimage_convert.o costella_esimage_magic.o
costella_huffman_1.o costella_huffman_2.o costella_huffman_3.o
costella_huffman_4.o costella_image.o costella_image_chrominance.o
costella_image_convert.o costella_image_simage.o
costella_image_simage_convert.o costella_jpc.o costella_quadword.o
costella_quantize.o costella_simage.o costella_simage_convert.o
costella_simage_magic.o costella_sort.o costella_wrap.o

jpc_to_bmp: jpc_to_bmp.o costella_base.o costella_bitstream.o
costella_bmp.o costella_eimage.o costella_eimage_convert.o
costella_eimage_esimage.o costella_eimage_esimage_convert.o
costella_esimage.o costella_esimage_convert.o costella_esimage_magic.o
costella_huffman_1.o costella_huffman_2.o costella_huffman_3.o
costella_huffman_4.o costella_image.o costella_image_chrominance.o
costella_image_convert.o costella_image_simage.o
costella_image_simage_convert.o costella_jpc.o costella_quadword.o
costella_quantize.o costella_simage.o costella_simage_convert.o
costella_simage_magic.o costella_sort.o costella_wrap.o

2) Please make sure the thing works:

thor@rusime04:~/src/jclear> bmp_to_jpc s.bmp s.jpc 163813

Costella library error tree:
----------------------------
CostellaJpcOutAll: Writing scale
CostellaJpcOutScale: Writing out Y channel
CostellaJpcChannelOutScale: Quantizing delta image
CostellaJpcChannelOutDownquantize: Starting next pass for
COSTELLA_QUANTIZE_OUT object
CostellaQuantizeOutStartNextPass: Creating new Huffman-4 object for
quantized level trial
CostellaHuffman4OutNew: Creating Huffman-3 object (message)
CostellaHuffman3OutNew: Creating standard Huffman-2 mode object
CostellaHuffman2OutNew: Creating output Huffman-1 object
CostellaHuffman1OutNew: Allocating frequency table

ERROR: Could not write out the JPC file.

thor@rusime04:~/src/jclear> uname -a
Linux rusime04 2.6.18.8-0.3-default #1 SMP Tue Apr 17 08:42:35 UTC 2007
x86_64 x86_64 x86_64 GNU/Linux

Ehem. That's not very usable in this way.

Happy debugging,

Thomas

He's Late

2007-11-23, 3:56 am

On Fri, 23 Nov 2007 17:15:41 +0900, Miles Bader <miles@gnu.org> wrote:

>Industrial One <industrial_one@hotmail.com> writes:
>
>Microsoft has a verrrry bad record of foisting really crappy data
>formats on the world.
>
>Previously they tried to foist a not-very-good HDR image format called
>"scRGB". Here's a detail description of HDR encodings that discusses
>some of the problems with scRGB:
>
> http://www.anyhere.com/gward/hdrenc/hdr_encodings.html
>
>"HD photo" seems to incorporate scRGB in some form, which doesn't bode
>well for its suitability as an HDR encoding...
>
>[Microsoft, of course, doesn't really care, as long as they own the
>patents to everything, they're happy whether or not the result sucks.]
>
>
>Well at least that's one things most people can agree on: Vista sucks.
>
>-Miles

I seem to have left out some important punctuation. Vista sucks, this is true,
but the HDR Photo(? I was in error in typing earlier) , HDPhoto format (*.hdp
and *.wdp filename extensions) has some very strong advantages, which got its
start through Vista. Full EXIF and IPTC support, wavelet compression, lossless
mode, 16-bit support, among other things. I've been using it once in a while
just to see how well it compresses and retains detail when comparing it to other
compression schemes. Not by using Vista, I won't go near that. By using
PhotoLine 32, a 32/64-bit editor which has already incorporated HDPhoto. PL32's
"for web" optimizer has a 3-spread panel where you can instantly compare
compression formats magnified against one another making it easy to see which is
to your advantage depending on the content of the image. For a new "standard"
that's eventually going to flood the scene, if for no other reason than all
those people who got stuck with Vista and used it, I can think of worse
"standards" that caught on. I'm looking forward to more applications supporting
this HDPhoto format now. I'd be using it today if noting more than a few
browsers supported it.

Thomas Richter

2007-11-23, 3:56 am

He's Late schrieb:

> I seem to have left out some important punctuation. Vista sucks, this is true,
> but the HDR Photo(? I was in error in typing earlier) , HDPhoto format (*.hdp
> and *.wdp filename extensions) has some very strong advantages, which got its
> start through Vista. Full EXIF and IPTC support, wavelet compression,

Pardon me, HDPhoto is not wavelet based. It uses an overlapped block
transform.

> lossless
> mode, 16-bit support, among other things. I've been using it once in a while
> just to see how well it compresses and retains detail when comparing it to other
> compression schemes.

Does it? I mean, seriously, I'd like to see examples where you believe
it does. Because either we're using a different HDPhoto here, or I do
something terribly wrong, both of which could be right, but it doesn't
seem to preserve detail very well here.

So long,
Thomas

John Costella

2007-11-23, 7:56 am

On Nov 23, 8:01 am, Phil Carmody <thefatphil_demun...@yahoo.co.uk>
wrote:
> John Costella <jpcoste...@hotmail.com> writes:
>
> Don't just say they're comparable - show us the data.
>
> Phil
> --
> Dear aunt, let's set so double the killer delete select all.
> -- Microsoft voice recognition live demonstration

Sorry Phil if I didn't make it clear. There are explicit comparisons
on the website
and you can also run the conversion programs on your own images if you
think the
standard test images are no longer appropriate.

John

John Costella

2007-11-23, 7:56 am

On Nov 23, 2:27 pm, Industrial One <industrial_...@hotmail.com> wrote:
> On Nov 22, 11:30 am, John Costella <jpcoste...@hotmail.com> wrote:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Awesome! Now... if I were you, when I'd announce a new technology I
> wouldn't yap my ass off about minor features nobody cares to know but
> state good reasons why it should replace previous JPEG by comparing
> sizes, quality, giving us a summary what JPEGClear has that JPEG or
> JPEG2000 doesn't. Follow?

I didn't post all that in the announcement because there are many
pages
on the website that do those comparisons.

It's difficult to know which of them are "minor" features; irrelevance
for one
developer might be a major factor for another.

> If the lossless engine is comparable to PNG then why the hell would we
> need your shit?

If you mean for lossless compression, then you would only want my shit
if you cared about fast, antialiased progressive downloads and/or
fast,
antialiased intrinsic thumbnails.

John

John Costella

2007-11-23, 7:56 am

On Nov 23, 4:18 pm, He's Late <betterl...@never.net> wrote:
> On Thu, 22 Nov 2007 19:27:48 -0800 (PST), Industrial One
>
> <industrial_...@hotmail.com> wrote:
>
>
> Doesn't matter, he's way too late. The new HD Photo 16-bit-support image format
> is already being supported in programs like PhotoLine 32. It's already doing all
> that stuff. Started by Vista, one of the few things that Gates did right.

As I've tried to make clear on my site, JPEG-XR/HD-Photo has some good
features, and JPEG-Clear is not trying to knock it off; the formats
are
complementary.

As far as I'm aware there are quite a few formats that support 16 (and
higher) bits
per channel. I'm just letting potential users know that (a) it isn't
restricted to just
8 bits per channel, and (b) it doesn't currently do more than 16 bits
per channel.

John

Douglas

2007-11-23, 7:56 am

"John Costella" <jpcostella@hotmail.com> wrote in message
news:51db890c-114a-4116-b2f4-3a04fc04b667@s12g2000prg.googlegroups.com...
> On Nov 23, 8:01 am, Phil Carmody <thefatphil_demun...@yahoo.co.uk>
> wrote:
>
> Sorry Phil if I didn't make it clear. There are explicit comparisons
> on the website
> and you can also run the conversion programs on your own images if you
> think the
> standard test images are no longer appropriate.
>
> John

You've hit a snag here John.
Certainly this format has the potential to change forever how images are
stored but... You can't see them!
I'd have thought the first step before release might have been a method of
viewing them in Windows Explorer and Internet explorer followed closely by
Photoshop and a few other editing programs.

Unless I've missed something on your rather cryptic web site... Even your
"examples" are in PNG format!!!!

Douglas

John Costella

2007-11-23, 7:56 am

On Nov 23, 6:43 pm, Thomas Richter <t...@math.tu-berlin.de> wrote:
> John Costella schrieb:
>
>
>
>
>
>
>
>
>
>
>
> First of all, I wouldn't call it JPEG unless it is coming from the JPEG.
> Strictly speaking, we don't have a trademark AFAIK, but still, I think
> it is confusing and I would recommend not to do this. I don't think we
> will come after you, but please, *don't do that*. Thanks!
>
> Second point is that I think you should definitely measure compression
> performance and publish test results on that. By performance I do not
> only mean PSNR, but other metrics as well: PQS, M-SSIM, VDP. If you need
> help on that, let me know.
>
> Third, as you may or may not know, JPEG currently has an open call for
> advanced image coding (AIC) where, if competitive, you may file your
> format for discussion in the JPEG and finally can standardize it. I
> highly recommend, though, to make measurements upfront.
>
>
> You may want to compare to state of the art, which is for example
> JPEG-LS. PNG is not a very reasonable lossless compressor.
>
>
>
> Have you considered implementing a smarter rate allocation, either
> a posteriori as in the EBCOT in JPEG2000, or an a priori algorithm that
> estimates the rate from the coefficient statistics? If it needs "passes"
> to find the rate, it looks like you can do better.
>
>
> As long as it is within the limits of JPEG2000, I wouldn't care as long
> as it is reasonably better than JPEG-XR.
>
>
> ...which is too bad.
>
>
> Please think about ICC profiles to support arbitrary color spaces.
> (Besides, there is no "the RGB" color space)
>
> So long,
> Thomas

Thanks Thomas. I knew I'd be hauled over the coals by the expert. I'm
surprised you haven't savaged me too badly (yet). :)

I agree that there is much to be measured, and the results I have
posted on my site are only a meagre start. But the fundamental problem
I have is that it's not clear how to perform fair comparisons. Let me
try
to explain.

This might not make sense until you've had a look at what I'm on about
with the format. It does lossless compression (pretty well) and lossy
compression (ok, but could probably be tweaked a lot more), but
that's
not the fundamental issue, which is that the format stores images in
a
top-down manner, in such a way that you get damn good progressive
download performance and antialiased thumbnails, as an intrinsic and
inherent part of the compressed file.

That then begs the question: what can I compare *that* to? Maybe
Adam7 for PNG. Maybe I pull out some progressive modes for
other formats. But if you've run the simulator I posted (sorry, the
simulator is Windows only),

http://www.assassinationscience.com...progressive.exe

then please tell me if JPEG-LS or the other formats can provide
comparable performance. (If so, then why aren't they being used
for web pages today?)

You'd then have to somehow quantify the question: How much is
this progressive downloading (or equivalently the thumbnails)
"worth" to you, relative to compression ratio? I doubt that it makes
sense to have a single number that answers this, for all users and
all
uses. In other words, how do you perform the sum

A * ( progressive performance ) + B * ( thumbnail performance )
+ C * ( compression performance ),

if you don't know A, B and C?

I guess what I'm saying is that you're used to measuring
apples and I'm giving you an orange, and it's not obvious to me
how you compare them. The "storage paradigm" is different.

As for improving the lossy compression parameter estimation,
I agree with you completely. The "multiple pass" method that
is in there at the moment is a poor one, but it is at least a
starting point. There is a significant amount of further research
that needs to be done, but I decided that I could not do it alone
(not least of which because I am colorbind).

I did my comparisons against PNG because that is widely
supported. As stated, the format is optimised for "consumers"
of images, especially images to be posted to web pages. I
agree that it would be good if I could compare it to every other
format on the planet, but there is only so much you can get
done on the commuter train every morning and evening. :)

I'll endeavour to do more testing, but at the moment I'm flat
out getting the animation/video format coded up. I decided
that it would be better to put out what I had, than spend three
years tying down every loose end (and indeed it is more
likely that I will get better ideas for improvement from others
than trying to continue on alone).

As for colorspaces, I take whatever RGB values are fed in
(if they are fed in as RGB) as naively as BMP or JPEG
does, with the expectation that the metadata will provide
sufficient information for the interpretation of those values.
I consciously decided to go for a "pragmatic" design
(concentrating on those things that are needed for most
boring real-world applications) rather than an "elegant"
design like JPEG-2000 (which appeals to me as a
physicist but, in practice, has many more degrees of
freedom to tie down).

In other words, I look at web pages full of JPEG and PNG
images, and photo folders full of JPEG images, and
aim to better than the status quo. I don't try to compete
with the J.P.E.G. (If I wanted a format designed by a
committee, I'd use one...)

I didn't realise that the J.P.E.G had asked for submissions.
That's an interesting idea, but at the moment I have enough
on my plate to keep me busy for a while. :)

Thanks
John

John Costella

2007-11-23, 7:56 am

On Nov 23, 7:28 pm, Thomas Richter <t...@math.tu-berlin.de> wrote:
> John Costella schrieb:
>
>
>
> Update to that:
>
> 1) Please add a Makefile. Here's one:
>
> CFLAGS = -Wall -pedantic -O3
> CC = gcc
> LDFLAGS = -lm
>
> all: bmp_to_jpc jpc_to_bmp
>
> bmp_to_jpc: bmp_to_jpc.o costella_base.o costella_bitstream.o
> costella_bmp.o costella_eimage.o costella_eimage_convert.o
> costella_eimage_esimage.o costella_eimage_esimage_convert.o
> costella_esimage.o costella_esimage_convert.o costella_esimage_magic.o
> costella_huffman_1.o costella_huffman_2.o costella_huffman_3.o
> costella_huffman_4.o costella_image.o costella_image_chrominance.o
> costella_image_convert.o costella_image_simage.o
> costella_image_simage_convert.o costella_jpc.o costella_quadword.o
> costella_quantize.o costella_simage.o costella_simage_convert.o
> costella_simage_magic.o costella_sort.o costella_wrap.o
>
> jpc_to_bmp: jpc_to_bmp.o costella_base.o costella_bitstream.o
> costella_bmp.o costella_eimage.o costella_eimage_convert.o
> costella_eimage_esimage.o costella_eimage_esimage_convert.o
> costella_esimage.o costella_esimage_convert.o costella_esimage_magic.o
> costella_huffman_1.o costella_huffman_2.o costella_huffman_3.o
> costella_huffman_4.o costella_image.o costella_image_chrominance.o
> costella_image_convert.o costella_image_simage.o
> costella_image_simage_convert.o costella_jpc.o costella_quadword.o
> costella_quantize.o costella_simage.o costella_simage_convert.o
> costella_simage_magic.o costella_sort.o costella_wrap.o
>
> 2) Please make sure the thing works:
>
> thor@rusime04:~/src/jclear> bmp_to_jpc s.bmp s.jpc 163813
>
> Costella library error tree:
> ----------------------------
> CostellaJpcOutAll: Writing scale
> CostellaJpcOutScale: Writing out Y channel
> CostellaJpcChannelOutScale: Quantizing delta image
> CostellaJpcChannelOutDownquantize: Starting next pass for
> COSTELLA_QUANTIZE_OUT object
> CostellaQuantizeOutStartNextPass: Creating new Huffman-4 object for
> quantized level trial
> CostellaHuffman4OutNew: Creating Huffman-3 object (message)
> CostellaHuffman3OutNew: Creating standard Huffman-2 mode object
> CostellaHuffman2OutNew: Creating output Huffman-1 object
> CostellaHuffman1OutNew: Allocating frequency table
>
> ERROR: Could not write out the JPC file.
>
> thor@rusime04:~/src/jclear> uname -a
> Linux rusime04 2.6.18.8-0.3-default #1 SMP Tue Apr 17 08:42:35 UTC 2007
> x86_64 x86_64 x86_64 GNU/Linux
>
> Ehem. That's not very usable in this way.
>
> Happy debugging,
>
> Thomas

Thanks, mate; I've tried to break it but my resources are limited.
(Would be much easier if my name was Bill Gates, but then
everyone would be throwing things at me.) Could you send me
the image that killed it? (Or are you telling me that it dies for all
images with that build?)

(Incidentally, that not-very-useful error tree actually says that the
program thought it ran out of memory, but that must be a bug;
I assume that s.bmp isn't some humungous image that won't
fit in memory...)

Does the Windows executable fail on the same image?

John

Thomas Richter

2007-11-23, 7:56 am

John Costella schrieb:

> Thanks, mate; I've tried to break it but my resources are limited.
> (Would be much easier if my name was Bill Gates, but then
> everyone would be throwing things at me.) Could you send me
> the image that killed it? (Or are you telling me that it dies for all
> images with that build?)

Pretty any image does that, see below for what I believe the problem is.

> (Incidentally, that not-very-useful error tree actually says that the
> program thought it ran out of memory, but that must be a bug;
> I assume that s.bmp isn't some humungous image that won't
> fit in memory...)
>
> Does the Windows executable fail on the same image?

No idea, no windows here.

Update to the update: It works with the -m32 switch, which means you
need to check the following issues:

"long" is not 32bit. It is what the compiler assigns to it. If you need
32 bit, int32_t is what you want, or an autoconfig script. For linux and
gcc, "long" is 64 bit.

Pointers aren't 32bit wide either. If you convert from a pointer to an
integer or back, make sure the corresponding integer type is wide
enough. ptrdiff_t for example, is wide enough to hold the difference
between any two pointers, so this might be the thing to check.

HTHH,
Thomas

John Costella

2007-11-23, 7:56 am

On Nov 23, 9:49 pm, "Douglas" <j...@the.group> wrote:
> "John Costella" <jpcoste...@hotmail.com> wrote in message
>
> news:51db890c-114a-4116-b2f4-3a04fc04b667@s12g2000prg.googlegroups.com...
>
>
>
>
>
>
>
>
> You've hit a snag here John.
> Certainly this format has the potential to change forever how images are
> stored but... You can't see them!
> I'd have thought the first step before release might have been a method of
> viewing them in Windows Explorer and Internet explorer followed closely by
> Photoshop and a few other editing programs.
>
> Unless I've missed something on your rather cryptic web site... Even your
> "examples" are in PNG format!!!!
>
> Douglas

Mate, are you pulling my leg, or are you serious? :)

The "examples" have all been converted to PNG format so that you can
see
them on the web site. I can't very well put

<img src="dog.jpc">

on my web site when no browser on the planet today knows what a
".jpc"
image is.

The progressive download simulator shows you what you would see in a
web browser if it *did* support the format. Likewise, the lossy
compression
comparer shows you actual results.

I think it would be possible to create a Photoshop plugin (if I had
the time
to dedicate to it), but it's not clear to me how I could "insert"
support for
a new image format into all the other scores of image editing and
viewing applications that are out there.

Likewise, I could put out a ".jpc viewer" app for Windows, but the
progressive
download simulator already lets you see what one of these files looks
like "in action" anyway.

I guess my intentions are not clear on the site ... I don't want to
compete
with Adobe or Microsoft or Mozilla; I'm giving this format out in the
hope
that others will pick it up and add it to the image formats they
currently
support. It's not supposed to stand out there by itself.

Obviously it wouldn't be too useful for web pages unless all the
major
browsers supported it, and I understand that that would be a difficult
thing to achieve. But I'm hopeful that the progressive download
benefits
(and the fact that it is completely unencumbered for either
proprietary
or open-source use) will help it get through the quagmire that has
killed
off just about everything else (except GIF, JPEG and PNG).

John

John Costella

2007-11-23, 7:56 am

On Nov 23, 11:02 pm, Thomas Richter <t...@math.tu-berlin.de> wrote:
> John Costella schrieb:
>
>
> Pretty any image does that, see below for what I believe the problem is.
>
>
>
> No idea, no windows here.
>
> Update to the update: It works with the -m32 switch, which means you
> need to check the following issues:
>
> "long" is not 32bit. It is what the compiler assigns to it. If you need
> 32 bit, int32_t is what you want, or an autoconfig script. For linux and
> gcc, "long" is 64 bit.
>
> Pointers aren't 32bit wide either. If you convert from a pointer to an
> integer or back, make sure the corresponding integer type is wide
> enough. ptrdiff_t for example, is wide enough to hold the difference
> between any two pointers, so this might be the thing to check.
>
> HTHH,
> Thomas

:( Damn. I thought I made everything portable, provided that long is
*at least* 32 bits, not just 32 bits. And nowhere does a difference
in
pointers need to be longer than 32 bits. (Long story.)

The costella_types.h file loads the .h file that maps all the types to
the
ones I use internally. The one it loads by default,
costella_types_ansi.h,
should have worked regardless, even if it is wasteful of memory.

Strange. I was sure I had compiled it on a 64-bit compiler (gcc on
linux).
Maybe I invoked the wrong one. (Beating head against table...)

Many thanks, Thomas, for going the extra mile and tracking this one
down for me. It is an embarrassing bug, and I'll obviously try to get
a
version 1.01 out there ASAP when I figure out where the logical error
is. I have a very good idea of where that is, and I suspect you have
hit the nail on the head with ptrdiff_t.

Cheers
John

(No Windows at all? Ah, I'd love to be able to say that I've kicked
that addiction...)

Thomas Richter

2007-11-23, 7:56 am

John Costella schrieb:

> This might not make sense until you've had a look at what I'm on about
> with the format. It does lossless compression (pretty well) and lossy
> compression (ok, but could probably be tweaked a lot more), but
> that's
> not the fundamental issue, which is that the format stores images in
> a
> top-down manner, in such a way that you get damn good progressive
> download performance and antialiased thumbnails, as an intrinsic and
> inherent part of the compressed file.

Of course, the natural respond to this question then would be: If you
want progression by resolution and a very good compression performance,
pick JPEG2000 which does that. (-: Actually, flexibility is probably the
main argument why you want something beyond JPEG. For *this* type of
target application, that is.

> That then begs the question: what can I compare *that* to? Maybe
> Adam7 for PNG. Maybe I pull out some progressive modes for
> other formats. But if you've run the simulator I posted (sorry, the
> simulator is Windows only),
>
> http://www.assassinationscience.com...progressive.exe
>
> then please tell me if JPEG-LS or the other formats can provide
> comparable performance.

Ok, compiled this with -m32 on my machine, now that it works, and
applied that to the shipbig test image. I assume that with no further
options, I get lossless compression?

Ok, here we go:

The original:
thor@rusime04:~/src/jclear> ls -l s.bmp
-r--r--r-- 1 thor users 3932214 23. Nov 09:21 s.bmp

With your compressor:

thor@rusime04:~/src/jclear> bmp_to_jpc s.bmp s.jpc
thor@rusime04:~/src/jclear> ls -l s.jpc
-rw-r--r-- 1 thor users 3072488 23. Nov 13:09 s.jpc

With jpeg2000, lossless (thus, the "-l" switch):

thor@rusime04:~/src/jclear> transcoder -l s.bmp s.j2k
thor@rusime04:~/src/jclear> ls -l s.j2k
-rw-r--r-- 1 thor users 2246765 23. Nov 13:09 s.j2k

With jpeg-ls (aka "loco"):

thor@latraviata:~/src/jpeg_ls_v2.2/Encoder> locoe
/store/pics/shipbig/shipbig_o.ppm -os.loco

========================================
=====
SPMG/JPEG-LS COMPRESSOR V.2.1
========================================
=====
These programs are Copyright (c) University of British Columbia.
All rights reserved. They may be freely redistributed in their
entirety provided that this copyright notice is not removed.
They may not be sold for profit or incorporated in commercial
programs without the written permission of the copyright holder.
Each program is provided as is, without any express or implied
warranty, without even the warranty of fitness for a particular
purpose.

========================================
=================
THIS SOFTWARE IS BASED ON HP's implementation of jpeg-ls:
========================================
=================
(c) COPYRIGHT HEWLETT-PACKARD COMPANY, 1995-1999.

Input file: /store/pics/shipbig/shipbig_o.ppm
Output file: s.loco
Image: cols=1280 rows=1024 alpha=256 comp=3 mode=1 (line intlv)
Parameters: T1=3 T2=7 T3=21 RESET=64 limit=23
Marker segment bytes: 37
Total bits out: 18331096 Symbols in: 3932160 4.662 bps (1.716 : 1)
Time = 1.330 secs : 2887 KSymbols/sec
thor@latraviata:~/src/jpeg_ls_v2.2/Encoder> ls -l s.loco
-rw-r--r-- 1 thor users 2291387 23. Nov 13:19 s.loco

On this specific image, jpeg2000 even seems to outperform loco, which
does not typically happen. I haven't picked the image for this specific
purpose, though.

For PNG:

thor@latraviata:~/src/jpeg_ls_v2.2/Encoder> pnmtopng
</store/pics/shipbig/shipbig_o.ppm >s.png
thor@latraviata:~/src/jpeg_ls_v2.2/Encoder> ls -l s.png
-rw-r--r-- 1 thor users 2423448 23. Nov 13:22 s.png

For HDPhoto:

thor@latraviata:~/src/HDPhoto> WmpEncApp -i
/store/pics/shipbig/shipbig_o.bmp -o s.wdp -c 9 -q 0
****************************************
***********************************
* Perf Report
****************************************
***********************************

Image Width = 1280, Height = 1024, total MegaPixels = 1.3 MP
m_ptEncDecPerf (excl I/O): 1802.458 milliseconds, 0.727185 MP/sec
m_ptEndToEndPerf (incl I/O): 1839.614 milliseconds, 0.712497 MP/sec
thor@latraviata:~/src/HDPhoto> ls -l s.wdp
-rw-r--r-- 1 thor users 2313737 23. Nov 13:23 s.wdp

> (If so, then why aren't they being used
> for web pages today?)

The popularity of a compression scheme is not in direct relation to its
compression performance, that's how the world is. There are many many
other factors much more relevant to market success. JPEG-LS aka Loco
also found its implementation in the Mars rover, IIRC: Very very low
complexity and very high efficiency for what it does. For the web, you
need a company that has sufficient impact to really drive a format.

> You'd then have to somehow quantify the question: How much is
> this progressive downloading (or equivalently the thumbnails)
> "worth" to you, relative to compression ratio? I doubt that it makes
> sense to have a single number that answers this, for all users and
> all
> uses. In other words, how do you perform the sum
>
> A * ( progressive performance ) + B * ( thumbnail performance )
> + C * ( compression performance ),
>
> if you don't know A, B and C?

Its the purpose of a metric to define those. But how to compare to
anything else if neither A,B or C is known? Basically, it means "I deny
to compare", but that's not an answer, it just denies the question.

> I guess what I'm saying is that you're used to measuring
> apples and I'm giving you an orange, and it's not obvious to me
> how you compare them. The "storage paradigm" is different.

Ok, so what's it? (Note well: I'm again playing devil's advocate, I
don't want to spoil the fun doing it) I've no problem at looking from
different angles at the problem, just that I need to know how.

> As for improving the lossy compression parameter estimation,
> I agree with you completely. The "multiple pass" method that
> is in there at the moment is a poor one, but it is at least a
> starting point. There is a significant amount of further research
> that needs to be done, but I decided that I could not do it alone
> (not least of which because I am colorbind).
>
> I did my comparisons against PNG because that is widely
> supported. As stated, the format is optimised for "consumers"
> of images, especially images to be posted to web pages. I
> agree that it would be good if I could compare it to every other
> format on the planet, but there is only so much you can get
> done on the commuter train every morning and evening. :)
>
> I'll endeavour to do more testing, but at the moment I'm flat
> out getting the animation/video format coded up. I decided
> that it would be better to put out what I had, than spend three
> years tying down every loose end (and indeed it is more
> likely that I will get better ideas for improvement from others
> than trying to continue on alone).
>
> As for colorspaces, I take whatever RGB values are fed in
> (if they are fed in as RGB) as naively as BMP or JPEG
> does, with the expectation that the metadata will provide
> sufficient information for the interpretation of those values.

As for JPEG, it only (historically) defined a toolbox, not a complete
file format. This is referred to as "codestream", where no color space
information is stored. JPEG2000 does the same, but in addition also
defines a file format where this is defined. It's probably a useful
paradigm to separate the issues.

> I consciously decided to go for a "pragmatic" design
> (concentrating on those things that are needed for most
> boring real-world applications) rather than an "elegant"
> design like JPEG-2000 (which appeals to me as a
> physicist but, in practice, has many more degrees of
> freedom to tie down).
>
> In other words, I look at web pages full of JPEG and PNG
> images, and photo folders full of JPEG images, and
> aim to better than the status quo.

Absolutely nothing wrong with that. All I say is "it's hard to do better
unless you know what 'better' means".

>I don't try to compete
> with the J.P.E.G. (If I wanted a format designed by a
> committee, I'd use one...)
>
> I didn't realise that the J.P.E.G had asked for submissions.
> That's an interesting idea, but at the moment I have enough
> on my plate to keep me busy for a while. :)

Please, do! As said above, I absolutely don't want to spoil the fun
(it is fun! for me it is, at least), so please keep working. I'm just
providing a "reality check" here. Don't be disappointed, there's much to
discover, and keep working.

So long,
Thomas

Thomas Richter

2007-11-23, 7:56 am

John Costella schrieb:

> :( Damn. I thought I made everything portable, provided that long is
> *at least* 32 bits, not just 32 bits. And nowhere does a difference
> in
> pointers need to be longer than 32 bits. (Long story.)

Don't be afraid, you're not the first falling into this trap.

> (No Windows at all? Ah, I'd love to be able to say that I've kicked
> that addiction...)

I do, but not in *this* office. I need to walk over to another one, but
I'm too lazy to do this right now.

So long,
Thomas

John Costella

2007-11-23, 7:56 am

On Nov 23, 11:28 pm, Thomas Richter <t...@math.tu-berlin.de> wrote:
>
>
> Ok, so what's it? (Note well: I'm again playing devil's advocate, I
> don't want to spoil the fun doing it) I've no problem at looking from
> different angles at the problem, just that I need to know how.

Thomas,

(Yes, please do be Devil's Advocate; we learn nothing without it!)

You are asking for a description of what the paradigm is?

The best way to see it is to wait until you walk to your other
office :)
and run the progressive download simulator. But the idea is very
simple. It is based on the "magic kernel" that does an obscenely
good job of upsampling and downsampling, given its simplicity.

The original image is repeatedly downsampled by a factor of two,
using the "magic kernel", until it is just a single pixel. Then you
send that single pixel. Upsample it one scale (one power of two)
and encode the difference between the upsampled and the
original, at that scale. Repeat until you get to the full image.

This would be of no use if it wasn't for that magic kernel. That's
what makes the "thumbnail" images look good, when they are
enlarged back to the "full" size, and what ensures that the
"thumbnails" are themselves antialiased.

So what you effectively get is an image that has been rearranged
optimally for either progressive download or the extraction of
thumbnails. (These are really the same thing; you just upsample
the latter to get the former, using the magic kernel.)

The lossy compression simply quantises the "delta" image at
each scale (or more so for the larger scales). This isn't a very
intelligent way of throwing away information, which is why the
lossy compression performance is not great.

I suppose ... it might be possible to simply use a different form
of lossy encoding (one that works better :) on each "delta"
image, rather than this naive quantising of values, but still keep
the overall paradigm of downsampling all the way down using
the "magic kernel" at the start, and upsampling back again,
scale by scale. You'd then still have the excellent progressive
download and thumbnail properties, but might have a better
chance at getting a competitive compression ratio.

The question then, I guess, is what the "best" form of
lossy compression would be for each of the "delta" images?
What's the best available today, that isn't bogged down in
the patent quagmire (like JPEG-2000)?

I guess it would be possible to use plain old JPEG ... hmm,
that would be ironic.

Thanks again ... you're inspiring some interesting thoughts ...
John

John Costella

2007-11-23, 6:56 pm

Sachin Garg

2007-11-23, 6:56 pm

On Nov 23, 5:28 pm, Thomas Richter <t...@math.tu-berlin.de> wrote:
[snipped lots of results]
> On this specific image, jpeg2000 even seems to outperform loco, which
> does not typically happen. I haven't picked the image for this specific
> purpose, though.

Thats probably because of color decorrelation. Jpeg2000 does it, Jpeg-
ls 'standard' leaves it to application and the UBC implementation you
used doesn't.

But thats just a guess.

Sachin Garg [India]
www.sachingarg.com | www.c10n.info

Martin Brown

2007-11-23, 6:56 pm

On Nov 23, 11:43 am, John Costella <jpcoste...@hotmail.com> wrote:
> On Nov 23, 6:43 pm, Thomas Richter <t...@math.tu-berlin.de> wrote:
>
>
>
>
>
>
>

There are some interesting ideas here. But you don't do yourself any
favours by referring to the 1,3 kernal as magical. More detail on the
mathematics underpinning the basic encoding and decoding method would
be useful.

Although there are other competing compression methods with already
established codecs. You might want to consider producing a plugin for
Internet Exploder so that people can download images in this new
format and see it in action. I think at this stage in the game that is
your path of least resistance (as happened with PNG early on).

JPEG is not quite as bad at compressing as your comparison program
would seem to suggest. At very high compression and low bit rates you
have to allow JPEG to use custom optimised Huffman tables in the
entropy encoding step rather than the generic defaults. Otherwise the
attainable compression is severely limited and the test is unfair.
JPEG still can't manage 1000:1 on most material but it gets a fair bit
closer.

A quick check using your sample point_nepean.bmp suggests the ultimate
compression limits are:

Default Huffman: 6524 = 144x
Optimised Huffman: 3056 = 306x
Optimised Huffman, Qtable: 2379 = 393x (already mostly monochrome)
Monochrome Optimised: 1587 = 590x

A couple more bytes could be shaved off by tweaks to the file layout.
IJG codec isn't quite optimal. And JPEG spec doesn't allow redundant
null Huffman codes to be trimmed out.
[color=darkred]
> This might not make sense until you've had a look at what I'm on about
> with the format. It does lossless compression (pretty well) and lossy
> compression (ok, but could probably be tweaked a lot more), but
> that's
> not the fundamental issue, which is that the format stores images in
> a
> top-down manner, in such a way that you get damn good progressive
> download performance and antialiased thumbnails, as an intrinsic and
> inherent part of the compressed file.

> That then begs the question: what can I compare *that* to? Maybe

To be fair you should probably compare it against progressive JPEG.

> Adam7 for PNG. Maybe I pull out some progressive modes for
> other formats. But if you've run the simulator I posted (sorry, the
> simulator is Windows only),
>
> http://www.assassinationscience.com...r/code/win_g...
>
> then please tell me if JPEG-LS or the other formats can provide
> comparable performance. (If so, then why aren't they being used
> for web pages today?)

Lossless is usually overkill for most web applications. JPEG-LS tends
to get used for medical or scientific imaging where true lossless high
compression is important to avoid introducing artefacts into
diagnostic images.

> You'd then have to somehow quantify the question: How much is
> this progressive downloading (or equivalently the thumbnails)
> "worth" to you, relative to compression ratio?

It might be interesting if the thumbnail could be delivered by a
server offering only the first part of the master file.

Regards,
Martin Brown

Thomas Richter

2007-11-23, 6:56 pm

John Costella schrieb:

> Thomas,
>
> I was unable to replicate the bug, even ensuring 64-bit code by using -
> m64 on 64-bit linux (gcc 4.1.0 suse linux). :(

You need to give it a target size to fail.

> However, could you substitute the file
> http://www.assassinationscience.com...ostella_types.h
> for costella_types.h and tell me if it still fails on your machine?

thor@latraviata:~/src/jclear> bmp_to_jpc /store/pics/shipbig/shipbig_o.bmp s.jpc 10000

Costella library error tree:
----------------------------
CostellaJpcInitialize: Initializing
CostellaQuantizeInitialize: Creating tables
CostellaQuantizeCreateTables: Allocating downquantization array

ERROR: Couldn't initialize costella libraries.

Further problems is that in the header you define

typedef unsigned long COSTELLA_UD;
typedef signed long COSTELLA_SD;

which should possibly be "int", and that you use the annex "UL" if you mean "32 bit unsigned", not "unsigned long".

Greetings,
Thomas

Thomas Richter

2007-11-23, 6:56 pm

John Costella schrieb:
> On Nov 23, 11:28 pm, Thomas Richter <t...@math.tu-berlin.de> wrote:
>
>
> Thomas,
>
> (Yes, please do be Devil's Advocate; we learn nothing without it!)
>
> You are asking for a description of what the paradigm is?
>
> The best way to see it is to wait until you walk to your other
> office :)
> and run the progressive download simulator. But the idea is very
> simple. It is based on the "magic kernel" that does an obscenely
> good job of upsampling and downsampling, given its simplicity.
>
> The original image is repeatedly downsampled by a factor of two,
> using the "magic kernel", until it is just a single pixel. Then you
> send that single pixel. Upsample it one scale (one power of two)
> and encode the difference between the upsampled and the
> original, at that scale. Repeat until you get to the full image.

In which sense is that different from what a wavelet does? It creates various scales
of the image and stores the difference of the upscaled image to the true image in
something called the "hi-pass" which is then entropy-coded.

> So what you effectively get is an image that has been rearranged
> optimally for either progressive download or the extraction of
> thumbnails. (These are really the same thing; you just upsample
> the latter to get the former, using the magic kernel.)

Clear enough, same thing for JPEG2000.

> The question then, I guess, is what the "best" form of
> lossy compression would be for each of the "delta" images?
> What's the best available today, that isn't bogged down in
> the patent quagmire (like JPEG-2000)?

I think if you look long and careful enough, you'll also find patents
in your compressor. There are too many of them. There are patents
for example that claim that if you want to transmit A and B, you can also
transmit A and B-A, and this would apply to your compression scheme (any many others).

Similar to a lot of patents, this one has never been challenged in a court, and probably
wouldn't stand a test, but... well. For ISO, we have *at least* companies that
allow royalty-free access to their technology, and that are strong enough to fight for
their technology in court.

> I guess it would be possible to use plain old JPEG ... hmm,
> that would be ironic.

Same patent problem here, see the Forgent patent. You cannot really do anything in this
field without trapping on a patent - it is a minefield. Even the most hilarious and obvious
techniques are patented - at least that's my impression.

So long,
Thomas

Marco Al

2007-11-23, 6:56 pm

Miles Bader wrote:

> http://www.anyhere.com/gward/hdrenc/hdr_encodings.html

Slightly out of date, they have added 32 bpc floating point.

Marco

aruzinsky

2007-11-23, 6:56 pm

On Nov 22, 12:30 pm, John Costella <jpcoste...@hotmail.com> wrote:
> 12:30 pm CST, November 22, 2007
> ...
> * Progressive, high-quality, antialiased decoding of any image, where
> the image simply gets "sharper" as more of it is downloaded.
> ...
>
> John Costella

That is a negative. All images should be their own progress bar by
downloading top to bottom (or vice versa). Otherwise, when I see an
image on the web that looks like crap, instead of waiting like a fool
to see if it "simply gets sharper as more of it is downloaded," I am
going to go to another web page.

Also, I.E. often stops downloading an image in the middle and clicking
the refresh button or "show image" doesn't help. Mix that with your
scheme and you really have disaster. Ever hear the saying, "The road
to hell is paved with good intentions."?

Industrial One

2007-11-23, 6:56 pm

On Nov 23, 1:25 am, Thomas Richter <t...@math.tu-berlin.de> wrote:
> Industrial One schrieb:
>
>
>
>
> "We" as in "the JPEG". Yes, I'm a JPEG member. (To be precise, of the
> "SC29/WG1" committee of the ISO, whose slang name happens to be "JPEG").
>
> So long,
> Thomas

The Matrix is a small world after all...

What was your role in the development of JPEG? Were you a key member?

The Spider Formally Seated Next To Little Miss Muf

2007-11-23, 6:56 pm

"He's Late" <betterlate@never.net> wrote in message
news:3dock3pa8ct88ldqie30llqapt29leatj6@
4ax.com...
> On Thu, 22 Nov 2007 19:27:48 -0800 (PST), Industrial One
> <industrial_one@hotmail.com> wrote:
>
>
> Doesn't matter, he's way too late. The new HD Photo 16-bit-support image
> format
> is already being supported in programs like PhotoLine 32. It's already
> doing all
> that stuff. Started by Vista, one of the few things that Gates did right.
>

Well, if PhotoLine 32 supports it then it must be the best thing. Right,
PhotoLine 32 is just a bunch of crapware from a person that has no talent,
inteligence or class.

The Spider

--
If stupid was fruit, Washington D.C. would be an orchard!

John Costella

2007-11-23, 6:56 pm

Hi Martin,

> There are some interesting ideas here. But you don't do yourself any
> favours by referring to the 1,3 kernal as magical. More detail on the
> mathematics underpinning the basic encoding and decoding method would
> be useful.

Sorry, mate, I forgot to mention that I discussed this at length last
year on
this group. With the help of the good folk here, it became clear why
it does
a good job of upsampling and downsampling. I kept the "magic" tag only
because I have no other name for it. I suppose I should really call
it
"the fancy 1:3 chrominance upsampling kernel used by the Independent
JPEG Group", but perhaps IJG would get upset if they thought I was
using
their name to promote the new format.

> Although there are other competing compression methods with already
> established codecs. You might want to consider producing a plugin for
> Internet Exploder so that people can download images in this new
> format and see it in action. I think at this stage in the game that is
> your path of least resistance (as happened with PNG early on).

That's not a bad idea, but I don't know if I can do that alone. I've
only got
a couple of hours a day to spend. Thomas has made some intriguing
suggestions that makes me think that it might be possible to switch
over
to a different form of compression at the later "scales" (whether
lossless
or lossy), and end up with the best of both worlds: fast progressive
and
thumbnail, yet compression as good as the best JPEG standards can
provide. Others have suggested a Photoshop plugin. I've also got the
animated/video and "container" formats imminent. And now we've
received a Notice to Vacate by January 21, and my backyard is full of
weeds... :(

I'd also be very reluctant to focus on only one vendor (Microsoft, if
I
focused on IE; Adobe, if I focused on Photoshop). There are many
people who use other equally good products. That's why I think it
needs
to come from the vendors and/or open source communities themselves.

I admit that my web site is currently skewed because the sample GUI
applications are Windows, as are the sample executables for the
console
apps. But that's only because I'm still addicted to Windows (I blame
the
summers I worked at IBM), and that was the easiest environment for me
to produce some demonstration GUI apps. The library itself doesn't
care
what operating system it is compiled on.

>
> It might be interesting if the thumbnail could be delivered by a
> server offering only the first part of the master file.

Yes, this is exactly how it works already! There are several ways that
it
can be done:

1. You start pulling the entire file from the server, but once you get
the
thumbnail that is sufficient for your purposes, you break the
connection
and ignore the rest. For example, you might pull down file dog.jpc,
which might be (say) 300 kB, but after 1000 bytes you decide you have
enough (or, more likely, the user decides they have seen enough and
they click through).

2. The server uses the "container" format (sorry, not yet delivered)
in
which the different scales are already split off into different
physical
disk files. You then only pull down a sufficient number of scales for
what you want. For example, a web page might have 8 different
images and animations on it. The creator of the web page has used
the ".jhc" format to bundle them all together. (This is at some date
in the distant future, of course, assuming that web browsers
understand these new formats.) Let's say that the largest image or
animation width or height is 500 pixels. The web server then has ten
files for that web page, which might be something like (I'm making
these numbers up, based on experience so far with the .jpc format)

images_00.jhc 50 bytes Header info, and all the images and
animations to scale 0
images_01.jhc 100 bytes The deltas for all the images and animations
to scale 1
images_02.jhc 250 bytes The deltas for all the images and animations
to scale 2
....
images_09.jhc 1,500,000 bytes The deltas for all the images and
animations to scale 9

The HTML for the web page would need to contain some sort of new tag
that tells compatible browsers that all (or some) of the images and
animations
on the page can be downloaded progressively from images_XX.jhc. When
the
browser pulls down images_00.jhc, it finds out how large each image
is, whether it
is animated, etc., and gets the "zeroth approximation" for each one
(no more
"black chasms"). It then pulls down images01.jhc, which progressively
fills in all
images and animations to scale 1. Etc, until images_09.jhc has been
downloaded,
or the user clicks through to something else.

The progressive downloading would look like the simulator I posted,

http://www.assassinationscience.com...progressive.exe

I agree that it would be nice if I could insert this behavior into IE,
Firefox, ...
but realistically I don't think I'm nearly good enough a programmer to
be able to
carry it off (if, indeed, it is possible to embed such behavior so
deeply in the
browser's code at all). And, again, there is the time factor ...

Thanks, mate, for the thoughtful comments.
John

John Costella

2007-11-23, 6:56 pm

On Nov 24, 2:25 am, Thomas Richter <t...@math.tu-berlin.de> wrote:
> John Costella schrieb:
>
>
>
> You need to give it a target size to fail.

AHH. Thank you. I thought it failed universally. Now I can debug ...

> Further problems is that in the header you define
>
> typedef unsigned long COSTELLA_UD;
> typedef signed long COSTELLA_SD;
>
> which should possibly be "int", and that you use the annex "UL" if you mean "32 bit unsigned", not "unsigned long".

You mean in costella_types_ansi.h? No, they were purposefully 'long'
just in
case there was a weird compiler that was 32-bit but defined 'int' to
be 16 bits.
(OK, I am showing my age here.) This is the "ansi" version of the
header file
that is supposed to work on any ANSI C compiler.

I understand that this will be wasteful of memory. The idea is that,
if you were
building a release version for a particular app, then you'd include a
different
costella_types_<my_architecture>.h that typedefs each of the
COSTELLA_XX
types to exactly the right type for your compiler and architecture.
(You mentioned
int32_t in a previous post, but I don't believe that this is in the
ANSI standard?)

I use UL to make sure that integers longer than 16 bits are correctly
represented (probably unnecessary). As I noted above, there is not (as
far as
I know) any ANSI compliant way to specifically specify 32-bit
integers.

However, it is possible that I have missed some explicit type casts
somewhere.
I will chase down the bug with your updated information.

Thanks, mate.
John

John Costella

2007-11-23, 6:56 pm

> I think if you look long and careful enough, you'll also find patents
> in your compressor. There are too many of them. There are patents
> for example that claim that if you want to transmit A and B, you can also
> transmit A and B-A, and this would apply to your compression scheme (any many others).
>
> Similar to a lot of patents, this one has never been challenged in a court, and probably
> wouldn't stand a test, but... well. For ISO, we have *at least* companies that
> allow royalty-free access to their technology, and that are strong enough to fight for
> their technology in court.
>
>
> Same patent problem here, see the Forgent patent. You cannot really do anything in this
> field without trapping on a patent - it is a minefield. Even the most hilarious and obvious
> techniques are patented - at least that's my impression.

Yes, I try to make the point that it not entrapped by any *genuine*
patents,
as far as I know. Sure, someone could pull out a piece of paper that
shows
they convinced the Patent Office to give them a patent on A and B -
A;
I had an interesting net argument with a crackpot who had managed to
patent a perpetual motion motion. But it doesn't make the patents
genuine
or enforceable. (And didn't the Forgent patent get chucked out of
court?)

I'm lucky that I'm not trying to make any money out of this new
format. But
I do understand that those writing the browsers and image applications
have
to worry about the patent sharks. But the day they run scared from
idiots
claiming a patent for B - A is the day that technology grinds to a
halt.
Fortunately, most of the big players have big enough legal sections
to
deal with the extortionists, and fairly compensate those with valid
claims.

John

John Costella

2007-11-23, 6:56 pm

On Nov 24, 4:04 am, aruzinsky <aruzin...@general-cathexis.com> wrote:
> On Nov 22, 12:30 pm, John Costella <jpcoste...@hotmail.com> wrote:
>
>
>
> That is a negative. All images should be their own progress bar by
> downloading top to bottom (or vice versa). Otherwise, when I see an
> image on the web that looks like crap, instead of waiting like a fool
> to see if it "simply gets sharper as more of it is downloaded," I am
> going to go to another web page.
>
> Also, I.E. often stops downloading an image in the middle and clicking
> the refresh button or "show image" doesn't help. Mix that with your
> scheme and you really have disaster. Ever hear the saying, "The road
> to hell is paved with good intentions."?

Mate, are you serious? You'd rather wait five minutes for an image to
load
line-by-line, than get as high-quality a progressive download as
possible?
I think you might be in the minority.

Don't get me started about some of the weird and wonderful things that
IE does. I agree that it would be useful for it to *tell* you if it
aborted
downloading things because you clicked something (or went Back
from another page, and it didn't bother finishing the download). But
that's a qualm with Microsoft, not with the underlying image formats.

I also don't think you'd be "waiting like a fool". Have you run the
progressive
download simulator? You get a pretty good indication that things are
progressing (and, if not, the status bar and/or progress thermometer
and/or spinning thing at the top of the browser) should be a good
indicator.

Appreciate the Devil's Advocate, but I do think you're trying to take
the
piss out of me here... :)

John

John Costella

2007-11-23, 6:56 pm

I have added the -m32 switch to the GCC compile
scripts, and added a note to the download page,
until I repair the bug.

Thanks
John

Thomas Richter

2007-11-24, 6:56 pm

Industrial One schrieb:

> What was your role in the development of JPEG? Were you a key member?

That would be second generation, JPEG2000, not JPEG. I'm to a good degree responsible
for the part-4 conformance test streams, part-2 and part-9 corrections and
extensions, not so much part-1. Part-9 testing/conformance is currently one focus for me,
another one is JPEG-XR which I'm co-chairing, AIC-Eval (testing codecs for compression
performance), delegate from Germany, University of Stuttgart, and co-financed by Pegasus
Imaging, www.jpg.com.

So long,
Thomas

Thomas Richter

2007-11-24, 6:56 pm

John Costella schrieb:
> I have added the -m32 switch to the GCC compile
> scripts, and added a note to the download page,
> until I repair the bug.

Will fix it, temporarily. My hint would be to have a serious look into
"autoconf", which will help you to figure out which types to use on the target platform.
You're of course right by saying that an int need not to be 32 bit.

(In fact, it need not to be "bit" at all as ANSI-C could run on a "ternary" computer.
Even though this is only science fiction, in reality things like 36bit integers have been seen,
so beware.)

So long,
Thomas

Barry L. Wallis

2007-11-24, 6:56 pm

Thomas Richter wrote:
....
> (In fact, it need not to be "bit" at all as ANSI-C could run on a "ternary" computer.
> Even though this is only science fiction, in reality things like 36bit integers have been seen,
> so beware.)

I cut my teeth on Decsystem-10s and 20s. In fact, I still have a t-shirt
that says, "If your computer has less than 36 bits, you're not playing
with a full DEC."

--
- Barry

Kenneth Sloan

2007-11-24, 6:56 pm

Barry L. Wallis wrote:
> Thomas Richter wrote:
> ...
>
> I cut my teeth on Decsystem-10s and 20s. In fact, I still have a t-shirt
> that says, "If your computer has less than 36 bits, you're not playing
> with a full DEC."
>

If DEC had paid more attention to proper grammar and usage perhaps they
would still be in business.

--
Kenneth Sloan KennethRSloan@gmail.com
Computer and Information Sciences +1-205-932-2213
University of Alabama at Birmingham FAX +1-205-934-5473
Birmingham, AL 35294-1170 http://KennethRSloan.com/

Rich

2007-11-24, 6:56 pm

On Nov 24, 12:30 pm, Kenneth Sloan <KennethRSl...@gmail.com> wrote:
> Barry L. Wallis wrote:
>
>
> If DEC had paid more attention to proper grammar and usage perhaps they
> would still be in business.
>
> --

Any guesses as to where the DEC Alpha would be relative to AMD and
Intel chips had they been able to keep going with it? It was already,
what, 7 years ahead of anything from Intel?

John Costella

2007-11-25, 3:56 am

On Nov 25, 1:47 am, Thomas Richter <t...@math.tu-berlin.de> wrote:
> John Costella schrieb:
>
>
> Will fix it, temporarily. My hint would be to have a serious look into
> "autoconf", which will help you to figure out which types to use on the target platform.
> You're of course right by saying that an int need not to be 32 bit.
>
> (In fact, it need not to be "bit" at all as ANSI-C could run on a "ternary" computer.
> Even though this is only science fiction, in reality things like 36bit integers have been seen,
> so beware.)
>
> So long,
> Thomas

That's the more conventional way to do it, but I
prefer a simpler approach, since all this is only
proof of concept anyway; anyone that would
incorporate the library is likely to set all the
settings in the header file (if not tweak it or
razor gang it even more, if they wanted to put
it into firmware, say). The files (minus the
bug) work fine for either a 32-bit or 64-bit compiler
without need to change anything, and that's the
way I intended it.

I have the bug fixed, thanks to your assistance,
but I want to do some more extensive testing
before releasing it. It was, as you highlighted,
a lapse of concentration when I wrote one part
of the quantisation table generator. I guess
that's the price you pay when you allow
emulation of 64-bit mode on a 32-bit compiler
and also allow that 32-bit emulation to be
compiled on a 64-bit compiler. :-) I kept my
wits about me except in one place, and that
lapse slipped through my testing.

HA ha ternary computer ... yeah well I'd have to
think carefully about what the bitwise operators
would mean ... are you sure that ANSI C really
supports them? (In SF world, of course ...)

Thanks again for your help in isolating the bug,
John

John Costella

2007-11-25, 3:56 am

On Nov 25, 4:52 am, Rich <rander3...@gmail.com> wrote:
> On Nov 24, 12:30 pm, Kenneth Sloan <KennethRSl...@gmail.com> wrote:
>
>
>
>
>
>
>
> Any guesses as to where the DEC Alpha would be relative to AMD and
> Intel chips had they been able to keep going with it? It was already,
> what, 7 years ahead of anything from Intel?

He he.

I don't mind 36 bits ... Thomas is only giving me nightmares
about systems that don't using binary at all ... :)

John

John Costella

2007-11-25, 6:56 pm

Thomas Richter

2007-11-26, 6:56 pm

John Costella schrieb:
> On Nov 25, 1:47 am, Thomas Richter <t...@math.tu-berlin.de> wrote:
>
>
>
> Thomas,
>
> The repaired version is now on the website. It should compile
> and run fine under either a 32-bit or 64-bit compiler. Let me
> know if you find any more holes in it. :)

Thanks, this works fine now. BTW, please, add a makefile. It makes thinks so much easier....

So long,
Thomas

John Costella

2007-11-26, 6:56 pm

On Nov 27, 3:30 am, Thomas Richter <t...@math.tu-berlin.de> wrote:
> John Costella schrieb:
>
>
>
>
>
>
>
>
>
> Thanks, this works fine now. BTW, please, add a makefile. It makes thinks so much easier....
>
> So long,
> Thomas

Thomas,

I *did* add the makefile, and removed the compile scripts, from the
web site.

http://www.assassinationscience.com.../jpegclear/code

Doesn't it appear on your browser? Maybe you need to reload that
page. (I didn't add all the .h dependencies to it yet, but plan to do
so shortly.)

Even though you prefer to run everything from the command line,
I still recommend that you run the Progressive Download Simulator

http://www.assassinationscience.com...progressive.exe

and take a look at how an image would appear in a browser. This
behaviour would be possible, no matter what other compression
method was "wrapped" by the format in the future.

Thanks again for helping pin down the bug.
John

David Thompson

2007-12-02, 6:56 pm

On Fri, 23 Nov 2007 16:18:47 -0800 (PST), John Costella
<jpcostella@hotmail.com> wrote:

> On Nov 24, 2:25 am, Thomas Richter <t...@math.tu-berlin.de> wrote:

> (You mentioned
> int32_t in a previous post, but I don't believe that this is in the
> ANSI standard?)
>
(Aside: The first de-jure standard for C was developed by/under ANSI,
but since 1990 it has actually been an ISO/IEC standard that is
adopted by ANSI (or nowadays apparently by INCITS on behalf of ANSI).
As well as by other appropriate national bodies like BSI and DIN. Many
people still call it an 'ANSI' standard anyway.)

The 'specified-width' types {,u}int{,_fast,_least}{N}_t and a few
others were added in the 1999 version commonly called C99 (formally
ISO/IEC 9899:1999). The 'exact-width' types like int32_t are not
strictly required to exist, if the underlying hardware doesn't support
them, but few machines nowadays don't. The _least variants are the
narrowest types at least N bits, and the _fast variants are the
'fastest' (in the judgement of the compiler writer) at least N bits,
and must exist at least for N=8,16,32,64.

C99 contained other bigger changes and full implementations are not
yet common, but this particular feature is easy enough to retrofit to
any C90 (or C95) implementation that has any 64-bit integers, and you
get standardized and thus supposedly known/understood naming.

If you want to see for yourself the actual (ANSI version of) C99 is
sold at webstore.ansi.org in pdf for personal use for USD30 by (at
least when I used it) US credit card only. You might try your national
standards organization; there have been reports that some sell at
better prices to their own nationals.

Alternatively, the base draft for the next version (colloquially C0x)
consists of C99 as amended by three (fairly minor) technical
corrigenda, and is available gratis from the committee website
www.open-std.org/jtc1/sc22/wg14 as committee document n1256.
(Officially you can't rely on committee documents to be _the_
standard, for example in a legal case, but if you only want to know
for your own pleasure they're good enough.) A mostly(?) updated
Rationale document, which describes and explains many (not necessarily
all) of the changes from prestandard 'K&R' C to C89/90, to C95, and to
C99, is also available at that website.

> I use UL to make sure that integers longer than 16 bits are correctly
> represented (probably unnecessary). As I noted above, there is not (as

No suffix is needed; from C89 (the first standard), the compiler will
choose a type wide enough for an integer constant (literal) (if it has
such a type at all). The exact list varies, since C99 adds 'long long'
types of at least 64 bits (and normally exactly 64).

If you are doing bitwise operations and shifts -- I haven't looked at
your code specifically, but most de/compression code does, a lot --
you probably want to be sure you use unsigned types and values. Some
aspects of bitwise and shifts for signed types are not fully defined
by the standard, and could vary, though again on most machines today
they don't. A signed type (at least int) 'meeting' a same-rank
unsigned across most operators (but not shifts) is converted to
unsigned, but you may need a U suffix for literals that stand alone,
and may want it for clarity even where it isn't strictly needed.
(Integer types strictly narrower than int, which means in most cases
u/s/char and some cases u/short, are separately promoted to u/int
BEFORE the other rules like across an operator are considered.)

> far as
> I know) any ANSI compliant way to specifically specify 32-bit
> integers.
>
Assuming you mean a (literal) value, if there is a 32-bit type then a
cast like (int32_t)1234 will be that type (or if there isn't such a
type, a compiler error, or in standardese 'required diagnostic').

And to answer another point downthread, C specifically requires
integer types to use binary representation, not ternary or anything
else; this basic requirement was in C89, and C99 expands on it to
specify signed types use 2's-complement, 1s'-complement, or
sign-and-magnitude. In practice nearly everyone today uses 2sC.
(There is no such requirement for floating-point in C; there are still
machines that use hex FP, and IBM is proposing changes to the 'IEEE'
floating-point standard, which similarly is now also an ISO/IEC
standard, to encourage use of decimal FP; C allows these.)

- formerly david.thompson1 || achar(64) || worldnet.att.net

John Costella

2007-12-03, 7:56 am

On Dec 3, 10:20 am, David Thompson <dave.thomps...@verizon.net> wrote:
> On Fri, 23 Nov 2007 16:18:47 -0800 (PST), John Costella
>
> <jpcoste...@hotmail.com> wrote:
>
> (Aside: The first de-jure standard for C was developed by/under ANSI,
> but since 1990 it has actually been an ISO/IEC standard that is
> adopted by ANSI (or nowadays apparently by INCITS on behalf of ANSI).
> As well as by other appropriate national bodies like BSI and DIN. Many
> people still call it an 'ANSI' standard anyway.)
>
> The 'specified-width' types {,u}int{,_fast,_least}{N}_t and a few
> others were added in the 1999 version commonly called C99 (formally
> ISO/IEC 9899:1999). The 'exact-width' types like int32_t are not
> strictly required to exist, if the underlying hardware doesn't support
> them, but few machines nowadays don't. The _least variants are the
> narrowest types at least N bits, and the _fast variants are the
> 'fastest' (in the judgement of the compiler writer) at least N bits,
> and must exist at least for N=8,16,32,64.
>
> C99 contained other bigger changes and full implementations are not
> yet common, but this particular feature is easy enough to retrofit to
> any C90 (or C95) implementation that has any 64-bit integers, and you
> get standardized and thus supposedly known/understood naming.
>
> If you want to see for yourself the actual (ANSI version of) C99 is
> sold at webstore.ansi.org in pdf for personal use for USD30 by (at
> least when I used it) US credit card only. You might try your national
> standards organization; there have been reports that some sell at
> better prices to their own nationals.
>
> Alternatively, the base draft for the next version (colloquially C0x)
> consists of C99 as amended by three (fairly minor) technical
> corrigenda, and is available gratis from the committee websitewww.open-std.org/jtc1/sc22/wg14as committee document n1256.
> (Officially you can't rely on committee documents to be _the_
> standard, for example in a legal case, but if you only want to know
> for your own pleasure they're good enough.) A mostly(?) updated
> Rationale document, which describes and explains many (not necessarily
> all) of the changes from prestandard 'K&R' C to C89/90, to C95, and to
> C99, is also available at that website.
>
>
> No suffix is needed; from C89 (the first standard), the compiler will
> choose a type wide enough for an integer constant (literal) (if it has
> such a type at all). The exact list varies, since C99 adds 'long long'
> types of at least 64 bits (and normally exactly 64).
>
> If you are doing bitwise operations and shifts -- I haven't looked at
> your code specifically, but most de/compression code does, a lot --
> you probably want to be sure you use unsigned types and