Java Help - Re: can Inflater be used to uncompress GZIP data?

Author

Re: can Inflater be used to uncompress GZIP data?

chattycow

2006-05-17, 7:06 pm

All of your ideas finally led me to the idea of extending the
InputStream and OutputStream classes to handle my own internal buffer.
This way I can write to the buffer(using OutputStream) while reading
from it(Using InputStream) in another thread. Once I create the
inputstream, I feed that into the GZIPInputStream class and start
uncompressing...the GZIPInputStream never sees an end to the stream, so
it doesn't have any problems.

Thanks for all your help,
----Dean.

Dale King

2006-05-18, 7:10 pm

chattycow wrote:
> All of your ideas finally led me to the idea of extending the
> InputStream and OutputStream classes to handle my own internal buffer.
> This way I can write to the buffer(using OutputStream) while reading
> from it(Using InputStream) in another thread. Once I create the
> inputstream, I feed that into the GZIPInputStream class and start
> uncompressing...the GZIPInputStream never sees an end to the stream, so
> it doesn't have any problems.

Note that the Java API already provides such an abstraction, the
PipedInputStream and PipedOutputStream combination.

But I think you are still making this too complicated. How are you
getting the data you want to pump into your buffer. Presumably you are
reading it on some other input stream somewhere. So why not eliminate
the middle man? Perhaps you should consider something like
SequenceInputStream which concatenates InputStreams.

You still have not adequately explained how the chunks of data are
getting to the machine to be decompressed. Why isn't is a single stream
of data?

--
Dale King

Eric Sosman

2006-05-19, 7:06 pm

chattycow wrote On 05/19/06 16:15,:
> [...]
> SequenceInputStream is fine if you already have all your streams
> identified, but I don't have all the data yet and I can't put it all in
> memory. SequenceInputStream will combine all the streams that you tell
> it, but it can't combine streams on the fly as you add them so it
> doesn't help.

I confess I haven't actually used it myself, but from
the documentation it certainly appears you can "dream up"
additional inputs as you go along. Use the

SequenceInputStream(Enumeration)

constructor, with an implementation of Enumeration that
decides (at run time) whether to produce another InputStream
or to call it quits.

--
Eric.Sosman@sun.com

Dale King

2006-05-22, 7:12 pm

chattycow wrote:
> The data is comming from a socket/network inputstream delivered to me
> ~32K at a time.
>
> No doubt pipedinput/output streams are an option. However, there is a
> few limitations with pipeinput/output streams...namely the static 1K
> buffer, the need to connect them, 2 classes as apposed to 1, and any
> other overhead associated with these...I'm sure there are more though.

I don't really see any limitations there. The 1K is not really a
limitation. The thread pushing data will have to block at some point.
Whether it blocks at 1K or 32K is not going to have any real impact that
I can see.

The fact that it is 2 classes makes sense because you have to
abstractions used from 2 different threads. Each end of the pipe gets
its own object.

> What I wrote is a very simply extension of inputstream and outputstream
> (~300 lines..w/ comments) connected by a single bytebuffer array.

Which is not as good of a design as putting it into 2 separate classes.

> I write blocks(32K at a time) to the output stream (that I extended).
> In another thread, I connect the GZIPInputStream to the inputstream(I
> extended) and start to uncompress.

But I'm really not trying to push you to the piped stream or your
implementation. I think both are the wrong idea. What you have is a
pull-push-pull design. You have one thread that is pulling data from the
server then pushing it to a buffer and you have another thread that is
reading from the GzipInputStream which is pulling from the buffer. I see
no reason to do all that. You should be able to that with just a pull
system where the GzipInputStream pulls from the server and eliminate the
middleman. But you have not been forthcoming on how you get the data
from the server to be able to advise you more completely.

> SequenceInputStream is fine if you already have all your streams
> identified, but I don't have all the data yet and I can't put it all in
> memory. SequenceInputStream will combine all the streams that you tell
> it, but it can't combine streams on the fly as you add them so it
> doesn't help.

As someone else pointed out you can set up your own enumeration so that
is not true.

> There are two features I'm trying to get:
> 1) The socket connection is also being used for general communication
> before and after the gzip file is transfered to transmit other items,
> informational messages, etc.. If I give the socket stream over to
> GZIPInputStream, when it completes it will close the entire connection.
> In addition, GZIPInputStream doesn't know it's done until it reads a
> "-1" from the inputstream...it assumes it's reading from a file
> connected to an input stream.
> 2) I want to be able to give a status of the file download while it's
> downloading via another monitoring thread. As far as I know, you can't
> do that with a stream that your already using for something else....so
> every 32K, I check to make sure the data is OK and count how much I've
> sent/received, how fast, etc.
>
> I'm sure everything I just said brought up a lot more questions? I
> posted back because I wanted others to see what my solution was so that
> they might have an answer when they come accross the same problem.

I still see nothing that justifies creating the other thread and using
the push-pull-push design. The simplest thing to do here is to create
your own implementation of InputStream that wraps your server
connection. That implementation would contain the logic that is
currently in your thread that is pulling the data from the server it's
just that it won't be done in a loop, but instead will respond to calls
for more data.

If you were to post the code that is reading from the server and pumping
it to your new class then prehaps I can show you how it would look.

> In any case, the problem is solved.

Solved quite incorrectly in my opinion.

--
Dale King

Sponsored Links