Home > Archive > Compression > May 2005 > libjpeg blows
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
|
|
|
|
IMHO, libjpeg represents all that is evil and unholy in software
engineering. Let me explain before you jump to the defense of this
defacto standard library and perhaps you'll agree that it deserves a
major overhaul.
Keep it simple stupid! I use to hate that saying when I first starting
programming long, long ago in a cubicle far, far way. All the old
farts were constantly spouting it off. I thought it simply went along
with their inherit resistance to change. If someone had shown me
libjpeg then, I probably would have thought it  and inovative in
the way it was engineered. But after working with various other
libraries and operating system APIs it occurs to me now that all that
ness and innovation really amounts to one thing: complexity.
Complexity where it isn't needed is an invitation to failure.
Take error handling as just one example of how overly complex libjpeg
is. In most APIs, the long accepted practice is to bubble up an error
to the application using the library and allow the application to
decide how to deal with it. This is typically done through the API's
return value or errno (in C/C++) or in some newer cases, exceptions.
In libjpeg you have to have an "ErrorManager" and the default
"ErrorManager" is unusable for all but the most simple of applications
because it writes a message to stderr and calls exit(). To most
applications, failure to load a jpeg that user incorrectly specified is
not the end of the world. Imagine your web browser exiting every time
it didn't find a jpeg on a page. "Stimpy, you fat bloated idiot, what
were you thinking!"
So you quickly realize that you need your own manager and you get to my
second beef with libjpeg: C is not object oriented; don't try and
pretend that it is. I have nothing against C++, or OOP, I use them
almost exclusively. OOP can actually lead to much simpler designs, but
choose the weapon right for you. Also worth noting is that
setjmp/longjmp != exception. In order to recapture control after an
error in libjpeg you have to use setjmp/longjmp which has side effects
that I won't go into.
This is just one example of the complexity of the libjpeg code, there
are plenty more. In the end, what value does this particular bit of
complexity add to the program or usage model? In my not quite humble
opinion: zero value added. You can argue until you are blue in the
face about how it works and that is the ultimate measure of code's
worth, but worth is always offset by cost and I'm sure that the cost of
writing the libjpeg code was more than if you just returned an error
code. Also the cost of writing programs that use it is more and the
cost of the anquish associated with using the lib is more and all that
cost detracts from it's worth.
If there are any profs or TAs reading these threads, please, as the
song goes, teach your children well.
-b
| |
| Guido Vollbeding 2005-05-09, 3:55 pm |
| b wrote:
>
> If there are any profs or TAs reading these threads, please, as the
> song goes, teach your children well.
First:
libjpeg was *not* written for the sake of profs or TAs!
libjpeg was written to establish a solid and platform-independent JPEG
implementation for universal practical application.
And by all means, there are few projects which were so successful in
achieving such goal as libjpeg was for the use of the JPEG standard!
> IMHO, libjpeg represents all that is evil and unholy in software
> engineering. Let me explain before you jump to the defense of this
> defacto standard library and perhaps you'll agree that it deserves
> a major overhaul.
No question, libjpeg deserves a major overhaul, and it *will* obtain
a major overhaul - don't worry!
But the time is not now, and actually libjpeg works quite well.
For some advanced features regarding future libjpeg development
see for example http://jpegclub.org .
> Keep it simple stupid!
I agree, but considering the oversized complexity in many
computing areas nowadays libjpeg is not the worst thing.
The problem is that actually the "establishment" still pushes
an *increasing* complexity and corresponding *decreasing*
understanding in most areas of computing!
It is a very demanding task for the remaining libjpeg maintainers
to *not* follow that evil path.
We *will* take care of the development, but we will not follow
evil paths as common today, but we will await the right moment
for coming activities...
libjpeg will, for good reason, remain in C, not C++.
The most important aspect for future libjpeg development is,
however, that it will require more responsibility for JPEG
development as a whole, not only implementation aspects!
The reason is that *official* JPEG development is today going the
wrong way: Increasing complexitity and decreasing understanding.
That mistake has to be corrected, and there is actually no
other force to correct this mistake than the Independent JPEG
Group, so the challenge is much greater than maintaining an
implementation library: We must, in the future, take the
responsibility and action for *proper* JPEG advantage and
better fundamental understanding of the JPEG process.
We have to work *against* established evil forces for this
goal, so please be patient...
Regards
Guido
| |
|
| Guido Vollbeding wrote:
> We have to work *against* established evil forces for this
> goal, so please be patient...
How do you throw an error out of it without exit() or longjmp()?
(Just curious...)
--
Phlip
[url]http://www.c2.com/cgi/wiki?Z Land[/url]
| |
| Guido Vollbeding 2005-05-09, 3:55 pm |
| Phlip schrieb:
>
> Guido Vollbeding wrote:
>
>
> How do you throw an error out of it without exit() or longjmp()?
You can use setjmp/longjmp. That is the standard exception handling
in C. It might not be optimal, but it worked for generations of
programmers, and I used it without major problem.
Regards
Guido
| |
|
| Guido Vollbeding wrote:
> You can use setjmp/longjmp. That is the standard exception handling
> in C. It might not be optimal, but it worked for generations of
> programmers, and I used it without major problem.
The OP complained in a way that seemed to speak of leaks. No prob: The
intervening functions probably don't use the heap.
--
Phlip
[url]http://www.c2.com/cgi/wiki?ZLand[/url]
| |
| Arthur J. O'Dwyer 2005-05-09, 3:55 pm |
|
On Mon, 9 May 2005, Phlip wrote in comp.programming:
>
> Guido Vollbeding wrote:
>
> How do you throw an error out of it without exit() or longjmp()?
I believe the OP (not Guido) stated that libjpeg takes an error
callback function, which itself may use exit() or longjmp.
What the OP was suggesting, and I half-agree with, is that libjpeg
/should/ handle errors the usual (or "naive," depending on your bias)
C way, which is to return some kind of error code, probably a small
negative integer with some standard #defined values. Then the client
code would simply check the return value of each libjpeg function
(which it really should be doing anyway, most likely) to see what
kind of error was encountered.
I half-disagree with the OP because I think it may be slightly easier
to write a nice error-code-using wrapper for a library function using
callbacks, than to write a nice callback-using wrapper for a library
function using error codes. But I haven't tested that assumption; it
may be entirely backwards.
-Arthur
| |
| Programmer Dude 2005-05-09, 3:55 pm |
| b writes:
> ....................... Also worth noting is that
> setjmp/longjmp != exception. In order to recapture control after an
> error in libjpeg you have to use setjmp/longjmp...
That, for me, puts libjpeg, as the Python would say, "Right Out!"
| |
| cr88192 2005-05-09, 8:55 pm |
|
"Arthur J. O'Dwyer" <ajo@nospam.andrew.cmu.edu> wrote in message
news:Pine.LNX.4.60-041.0505091129170.2491@unix46.andrew.cmu.edu...
>
> On Mon, 9 May 2005, Phlip wrote in comp.programming:
>
> I believe the OP (not Guido) stated that libjpeg takes an error
> callback function, which itself may use exit() or longjmp.
>
> What the OP was suggesting, and I half-agree with, is that libjpeg
> /should/ handle errors the usual (or "naive," depending on your bias)
> C way, which is to return some kind of error code, probably a small
> negative integer with some standard #defined values. Then the client
> code would simply check the return value of each libjpeg function
> (which it really should be doing anyway, most likely) to see what
> kind of error was encountered.
>
> I half-disagree with the OP because I think it may be slightly easier
> to write a nice error-code-using wrapper for a library function using
> callbacks, than to write a nice callback-using wrapper for a library
> function using error codes. But I haven't tested that assumption; it
> may be entirely backwards.
>
yeah, I agree with the op some as well, I prefer returned error codes.
in my case I don't know that much about libjpeg, I just beat together code
to "make it work", albeit imo in a sub-optimal way.
I more just complain that, I don't like dependency on external libs (which
has popped up, and causes me annoyance).
imo, I have come up with an acceptable alternative for my own use:
higher quality imo than jpeg (based on initial comparrison with greyscale
images where I got smaller sizes and also better looking images);
preserves higher detail in the uv channels due to not downsampling (I was
considering downsampling, but am now more considering using seperate
quantizer tables, and maybe quantizing the uv channels more harshly if
needed);
also preserves the alpha channel (a biggie for me);
....
of course, at present, it is a little larger than jpeg, which I will argue
is quite probably due to the not downsampling the uv (my thought is that
since textures are a lot more likely to be seen up close, harsh
downsampleing is probably not desired). thought: given I am using 4bpp for
alpha and dividing the values by 16, the max alpha is 240, thus all images
end up slightly transparent. may need to adjust scale or such, eg, so 15
maps to 255 and not 240, ... ammusingly enough, the new scale is an integer:
17...
of course, most of my textures are still jpeg, so probably jpeg will
continue to be used, with this more being an alternative to tga (it is
typically much smaller than tga, but it still has alpha channels).
of course, jpeg could probably sway me if it had an alpha channel...
still working on sound though, current attempt sounds like crap for the
bitrate...
| |
| Flavius Vespasianus 2005-05-13, 3:56 pm |
| "b" <bryanwilkerson@yahoo.com> wrote in news:1115484503.741755.303320
@o13g2000cwo.googlegroups.com:
>
> IMHO, libjpeg represents all that is evil and unholy in software
> engineering. Let me explain before you jump to the defense of this
> defacto standard library and perhaps you'll agree that it deserves a
> major overhaul.
In defense of LIBJPEG:
1. Without it, no one would be using JPEG these days (Note there is no
equivalent for JPEG 2000 and how many people are using it.)
2. The JPEG standard itself is highly defective. LIBJPEG in effect created
the "JPEG Standard" that we use today.
3. I suspect that when the author of JPEG started, he was wondering around
in the JPEG standard without a real target (the standard not giving one).
In short, I suspect he was trying to get something to work rather than to
get an engineered project.
4. It works.
| |
|
|
Flavius Vespasianus wrote:
> In defense of LIBJPEG:
>
> 1. Without it, no one would be using JPEG these days (Note there is
no
> equivalent for JPEG 2000 and how many people are using it.)
>
Absolutely agree. It's an interesting paradigm, for lack of a better
word only, that in the open source space, first to market engrains a
product even more than in the commercial space. I certainly don't
want to rewrite it and am forced to stick with it in effort to get my
code to "market" quickly. I'm very appreciative that libjpeg
exists.
> 2. The JPEG standard itself is highly defective. LIBJPEG in effect
created
> the "JPEG Standard" that we use today.
>
All the more reason why I don't want to rewrite it completely. :)
I think it could be revised to address my concerns.
> 3. I suspect that when the author of JPEG started, he was wondering
around
> in the JPEG standard without a real target (the standard not giving
one).
> In short, I suspect he was trying to get something to work rather
than to
> get an engineered project.
>
Disagree with you here. After reviewing the code and writing something
that uses it, my extremely humble and appreciative opinion is that it
is over engineered.
> 4. It works.
I've already addressed this point, but let me add that "It works"
is the battle cry of meritocracy in software engineering. I've had the
pleasure[??] to work closely with Microsoft over the years and I can
tell you that "It works" is gospel in Redmond. To overcome the blight
that has be perpetrated by believers of "It works" will require the
open source community to expend more effort to the church of "Get it
right". That will only happen if we have these discussions.
Now I'll step off of the pulpit and try to be constructive.
-bryan
| |
|
|
I was a little overdramatic in the original post and only mildly
constructive. It got the attention I wanted but the end doesn't
justify the means, so I apologize.
libjpeg doesn't "blow", but it could be better, less complicated and
easier to use. Here is a short list of what I would do if I could only
extend the Earth day by four hours:
1. Scrap the majority of the ErrorManager subsystem. As previously
mentioned setjmp/longjmp have side effects. In addition to screwing up
debuggers, this technique greatly increases the risk of leakage, more
so than C++ exception handlers or SEH subsystems.
2. Scrap the MemoryManager subsystem. Let the using application manage
their memory or use a library of their choosing to do this. The
jpeg_read_scanlines and jpeg_write_scanlines should be able to take a
buffer allocated by the user. I played with getting this to work by
casting the buffer parameter to a byte array that I allocated and I
couldn't get it to work. So, in order to get all of the RGB data
into a semi permanent buffer I can perform whole image transformations
on, I had to read one scan line at a time and then memcpy into my
buffer. Mine is not the only application that does it this way; CImage
and graphpite are two others and there are probably more. Not as
efficient if I had just given libjpeg a single buffer and it read all
of the scanlines at once directly into my buffer. I know from the
documentation that there is a way to do this with a custom Data
Destination Manager but that brings me to my next simplification...
3. Do away with Data Source and Data Destination Managers. If the user
controlled the buffer it would just pass it's preallocated buffer to
read and write functions. and do with it as it pleased.
Jpeg_read_scanlines and jpeg_write_scanlines should accept buffers for
both input and output. As it is, data managers are not documented very
well and examples are not easily located. As a result I suspect that
when someone does need to do as I did and stream the result of a
transformation over the network they end up writing the scanline to
memory and then rewriting the bits to a socket. I know there is a way
to do this with a custom data destination manager, but it's not
altogether intuitive as how this is done because...
4. Don't try and make C encapsulate data and operations by putting
function pointers in a data structure. I wouldn't go back and change
the internals of the library but I would consider reworking the API to
not force the user to deal with this. The C way of achieving the
original author's intent is to have a group of functions (module)
that all take a structure as a parameter that has the data in it. This
is much more intuitive than function pointers in a structure and would
be self explanatory to any C or C++ programmer who has ever had to deal
with similar systems such as pthreads or any of millions of other C API
groupings. Think callback function instead of overriding where you
want to allow the user to perform transformations etc. inline.
5. Man pages for each of the APIs por favor. The document is a nice
read on hows and whys but developers need references; i.e. quickly tell
all the parameters to jpeg_write_scanlines or the members of
jpeg_compress_struct.
That's it for now. My thanks to all,
-bryan
| |
|
| On 13 May 2005 12:02:13 -0700, "b" <bryanwilkerson@yahoo.com> wrote:
>2. Scrap the MemoryManager subsystem. Let the using application manage
>their memory or use a library of their choosing to do this. The
>jpeg_read_scanlines and jpeg_write_scanlines should be able to take a
>buffer allocated by the user. I played with getting this to work by
>casting the buffer parameter to a byte array that I allocated and I
>couldn't get it to work. So, in order to get all of the RGB data
>into a semi permanent buffer I can perform whole image transformations
>on, I had to read one scan line at a time and then memcpy into my
>buffer.
I've never had a problem getting jpeg_read_scanlines to dump data directly
to my frame buffer. How are you using it that you need memcpy?
>3. Do away with Data Source and Data Destination Managers. If the user
>controlled the buffer it would just pass it's preallocated buffer to
>read and write functions. and do with it as it pleased.
But then you have to manage a buffer, handle errors, etc. I think it's a
great idea for the library to have default internal handlers, *and* a way to
plug in your own handlers at will.
>Jpeg_read_scanlines and jpeg_write_scanlines should accept buffers for
>both input and output. As it is, data managers are not documented very
>well and examples are not easily located. As a result I suspect that
>when someone does need to do as I did and stream the result of a
>transformation over the network they end up writing the scanline to
>memory and then rewriting the bits to a socket. I know there is a way
>to do this with a custom data destination manager, but it's not
>altogether intuitive as how this is done because...
For one of my apps, I needed to decode a JFIF stream that was already in
memory. I did take a few hours to figure out how to set up a suitable data
source manager, but once done I could appreciate how flexible the interface
is. Could it be better? Probably, but simplicity at the expense of
flexibility seems like a poor bargain.
>4. Don't try and make C encapsulate data and operations by putting
>function pointers in a data structure. I wouldn't go back and change
>the internals of the library but I would consider reworking the API to
>not force the user to deal with this. The C way of achieving the
>original author's intent is to have a group of functions (module)
>that all take a structure as a parameter that has the data in it. This
>is much more intuitive than function pointers in a structure and would
I've done it both ways, storing function pointers in structs and passing
function pointers as parameters (a la qsort). Both have their uses, and
there's something to be said for keeping your argument list short.
I think it's misleading to use the word "intuitive" here. Nobody starts out
knowing how an API should look; your experiences just color your
expectations. I started out thinking that the libjpeg API had the most
convoluted, over-engineered, irritating interface I'd ever seen (certainly
nothing like the small, simple API to my GIF library). Once I spent some
time learning about libjpeg, though, I came to understand how this approach
places a lot of power in the hands of the application programmer.
The great thing, though, is that it would be fairly straightforward to
totally revamp the API to work just as you describe, simply by writing a set
of wrapper functions around the existing library.
>5. Man pages for each of the APIs por favor. The document is a nice
>read on hows and whys but developers need references; i.e. quickly tell
>all the parameters to jpeg_write_scanlines or the members of
>jpeg_compress_struct.
On this point you have my total agreement!
-Scott
| |
|
| b wrote:
>
> I was a little overdramatic in the original post and only mildly
> constructive. It got the attention I wanted but the end doesn't
> justify the means, so I apologize.
Think nothing of it: You deftly activated USENET's user interface.
What happens here stays here. ;-)
--
Phlip
[url]http://www.c2.com/cgi/wiki?ZLand[/url]
| |
| Flavius Vespasianus 2005-05-16, 3:55 pm |
| "b" <bryanwilkerson@yahoo.com> wrote in news:1116004966.928493.88320
@g47g2000cwa.googlegroups.com:
>
> Flavius Vespasianus wrote:
> around
> one).
> than to
> Disagree with you here. After reviewing the code and writing something
> that uses it, my extremely humble and appreciative opinion is that it
> is over engineered.
>
I will counter-disagree here. In "software engineering" you're supposed
to be able to take a written plan and turn it into code. The JPEG
standard is unimplentable.Not to rag on it too much but JPEG was the work
of academics who apparently didn't care whether it could be implemented.
You simply cannot create an "engineered" JPEG implementation from the
standard.
If you want understandable JPEG, you can always try the C++ and Delphi
source code at www.colosseumbuilders.com.
>
> I've already addressed this point, but let me add that "It works"
> is the battle cry of meritocracy in software engineering. I've had the
> pleasure[??] to work closely with Microsoft over the years and I can
> tell you that "It works" is gospel in Redmond. To overcome the blight
> that has be perpetrated by believers of "It works" will require the
> open source community to expend more effort to the church of "Get it
> right". That will only happen if we have these discussions.
>
True. But how many flaws do we find in LIBJPEG these days?
| |
| newstome@comcast.net 2005-05-16, 8:55 pm |
| In comp.compression Flavius Vespasianus <nl@nl.com> wrote:
> I will counter-disagree here. In "software engineering" you're supposed
> to be able to take a written plan and turn it into code. The JPEG
> standard is unimplentable.Not to rag on it too much but JPEG was the work
> of academics who apparently didn't care whether it could be implemented.
Except for the fact that "academics" had basically nothing to do with
the JPEG standard. It was a (purely?) industry effort.
The key contributors to the standard came from IBM, DEC, C-Cube, NTT,
Siemens, British Telecom, and a few other companies.
--
That's News To Me!
newstome@comcast.net
|
|
|
|
|