Home > Archive > Compression > January 2006 > Ruby - Zlib - Invalid Distance too far back
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Ruby - Zlib - Invalid Distance too far back
|
|
| chris.r.malek@gmail.com 2006-01-13, 6:56 pm |
| I hope someone in the group can tell me this is a simple example.
I have a pretty basic Ruby script where I am trying to decode and
string then inflate it. However Zlib is throwing a string error that I
can't track down the source.
The error when running this script is:
Ruby Version: 1.8.2
Zlib Version: 1.2.1
The error Message is:
"invalid distance too far back"
########################################
#################
require 'base64'
require 'zlib'
begin
f =3D File.open("c:/temp/string.txt","r")
fout =3D File.open("c:/temp/decodedString","w")
while (ln =3D f.gets)
begin
decodeString =3D Base64.decode64(ln)
fout.puts("Here comes the decoded string \n")
fout.puts(decodeString)
inflatedString =3D Zlib::Inflate.inflate decodeString
fout.puts("here comes the inflated string \n\n")
fout.puts(inflatedString)
rescue StandardError =3D> err
fout.puts("an exception occured \n #{err}\n")
print "an exception occured #{err}\n"
end #block under while loops
end #while loop
ensure
f.close
fout.puts("closing program from ensure")
fout.close
end #top begin section
########################################
#################
The string in the file I am operating on is:
eJztmF1vmzAUhv9KlJteZdgQCEjElWNcghSc1Jiq
ubKiLpoqZd1Uomn79zMfTY1Jq1VJ14ty5/O=
=AD+PseHk0dWILz8/X03+LV9LO5/ PEwv4BdwcYlClkvBMcsIT1ZC3ixzMqdcZmtGUHh1
v919FX9=
+bo=ADtmH5M3ZM5lhMXgbrcpiumQD1FI09UiiQZ7
tXE6JHPMh5YSb4Vc8lh2DFWo9AjmlPLjnqq=
PJZs9u=ADSxPZ7TyM3qtgo781PWxswinWFB5tcBx
11zxhFXH0cYrl6VXG7NciCXrptkskpxe5zQ=
ThtmoMhJG=ADQZVVLuUqny2SbG64ljFd1UBGcIr1
KteS8wi3McU0nMp8R5lAiJiTBbneGMypfz0=
iyWaqJqNuZoV=ADnxJaJbJhGUCM0I7E28eQ/ J8QbtjV4XZMqIMp9RwrPq51EJnKy2+RZv9BoXic=
fNQbO72CtB/IA4B=ADACAIAjcIrUbRsEPABtWGsXKf1TZ/KI95VI1f00wMETzsOEgNiqXVrDQKU=
QB91ZMmGCSidVVRV9o=AD4ItW7O4JwBOqzNafFZj
nRVtimE7EqW1c0RFWyHhiIli146vwRtKsiL=
e+txCLK4tDSBY1aVU0PTG=ADrr3I6q46sqtCKTXw
TKORmaCXE5S1PRSK7P0KIDyi1q347w2HE9F=
x5HGMLxGDi+/54Ig6MIO14wO=ADQPCYDyy7R7hz8wv9GznA/idlFdnz2/P76n82q7vfsj96/f89=
vyezq8T2O/6F/jF+xf0/Pb8nsav=AD+gGgM/Em//8VTt2/53iFO53f9av81of4I9vrST4zydbhs=
4T1ykeyv+QRmHM=3D
| |
| Mark Adler 2006-01-13, 6:56 pm |
| The zlib stream you sent, ending in base64 with "QRmHM=" is incomplete.
It stops in the middle of the compressed data.
mark
| |
| Jasen Betts 2006-01-14, 3:55 am |
| On 2006-01-13, chris.r.malek@gmail.com <chris.r.malek@gmail.com> wrote:
> I hope someone in the group can tell me this is a simple example.
>
> I have a pretty basic Ruby script where I am trying to decode and
> string then inflate it. However Zlib is throwing a string error that I
> can't track down the source.
how is the data compressed. zlib ssupports three formats,
raw deflate, zlib deflate, and gzip.
AIUI you need to initialise the inflater (you do in C)
and you don't seem to be doing that.
Bye.
Jasen
| |
| Chris Malek 2006-01-14, 6:55 pm |
| >Mark Adler: The zlib stream you sent, ending in base64 with "QRmHM=" is incomplete. It stops in the middle of the compressed data.
>Jasen Betts: how is the data compressed. zlib ssupports three formats,
This string is actually coming from a PeopleSoft Application Messaging
module. So I really don't have any way of knowing there exact method.
According to the documentation on thier website they send messages in
be "deflated and encoded in base64".
I did not see in the ruby documentation of where to set the compression
method. Is there a standard method that is mostly used? PeopleSoft
in general follows most web standards.
This is troubling.
| |
| Mark Adler 2006-01-14, 6:55 pm |
| The data you sent is a zlib stream, as identified by the first two
bytes. Ruby's zlib interface does allow you to select zlib, gzip, or
raw through the specification of window_bits to inflate.new (if it's
hooked up to zlib 1.2.x), but you don't need to mess with that for this
application, since the default assumes zlib streams.
Chris Malek wrote:
> This is troubling.
As I look at it more closely, there is an error in the compressed
stream. There may have a transmission or translation error at some
point.
mark
| |
| Jasen Betts 2006-01-15, 3:55 am |
| On 2006-01-14, Chris Malek <chris.r.malek@gmail.com> wrote:
>
>
> This string is actually coming from a PeopleSoft Application Messaging
> module. So I really don't have any way of knowing there exact method.
> According to the documentation on thier website they send messages in
> be "deflated and encoded in base64".
>
> I did not see in the ruby documentation of where to set the compression
> method. Is there a standard method that is mostly used? PeopleSoft
> in general follows most web standards.
I don't have ruby installed here, neither do I have access to the
documentation of the ruby-zlib API... as for web standards there's
standards and then there's the microsoft way.
post or email me some of the base 64 encoded data (3 or 4 lines would be
plenty) and I'll see which flavour it is.
Bye.
Jasen
| |
| Chris Malek 2006-01-17, 6:55 pm |
| Here is the XML message that I am really try to parse.
The <data> element is really the deflated and encoded.
Thanks for you help.
<?xml version="1.0"?><request version="1.1"><from node="PS_SAIDP"/><to
node="PSFT_EP"/><operations namespace="PublishSubscribe"
interface="PublishSubscribeSystem"><invoke member="Publish"><variable
type="object"
interface="Publication"><publication><publishingnode>PS_SAIDP</publishingnode><channel>NU_TRANSCRIPT_VOUCHER_SYNC</channel><publicationid>4</publicationid><subject>NU_TRANSCRIPT_VOUCHER_SYNC</subject><class>PSAPMSG</class><originatingnode>PS_SAIDP</origin
atingnode><publisher>CMALEK</publisher><publicationprocess>NU_SR0021</publicationprocess><publishtimestamp>2006-01-12T17:32:16.973000-0800</publishtimestamp><status>4</status><defaultdataversion>VERSION_1</defaultdataversion><dataversions><publicationdata
version><version>VERSION_1</version><data
encoding="base64(deflate)" encodedlength="808(605)"
length="4950"> eJztmF1vmzAUhv9KlJteZdgQCEjElWNcghSc1Jiq
ubKiLpoqZd1Uomn79zMfTY1Jq1VJ14ty5/O+PseHk0dWILz8/X03+LV9LO5/ PEwv4BdwcYlClkvBMcsIT1ZC3ixzMqdcZmtGUHh1
v919FX9+botmH5M3ZM5lhMXgbrcpiumQD1FI09Ui
iQZ7tXE6JHPMh5YSb4Vc8lh2DFWo9AjmlPLjnqqP
JZs9uSxPZ7TyM3qtgo781
PWxswinWFB5tcBx11zxhFXH0cYrl6VXG7NciCXrp
tkskpxe5zQThtmoMhJGQZVVLuUqny2SbG64ljFd1
UBGcIr1KteS8wi3McU0nMp8R5lAiJiTBbneGMypf
z0iyWaqJqNuZoVnxJaJbJhGUCM0I7E28eQ/ J8QbtjV4XZMqIMp9RwrPq51EJnKy2+RZv9BoXicf
NQbO72CtB/IA4BACAIAjcIrUbRsEPABtWGsXKf1TZ/KI95VI1f00wMETzsOE
gNiqXVrDQKUQB91ZMmGCSidVVRV9o4ItW7O4JwBO
qzNafFZjnRVtimE7EqW1c0RFWyHhiIli146vwRtK
siLe+txCLK4tDSBY1aVU0PTGrr3I6q46sqtCKTXw
TKORmaCXE5S1PRSK7P0KIDyi1q347w2HE9Fx5HGM
LxGDi+/54Ig6MIO14wOQPCYDyy7R7hz8wv9GznA/idlFdnz2/P76n82q7vfsj96/f89vyezq8T2O/6F/jF+xf0/Pb8nsav+
gGgM/Em//8VTt2/ 53iFO53f9av81of4I9vrST4zydbhs4T1ykeyv+QR
mHM=</data></publicationdataversion></dataversions></publication></variable></invoke></operations></request>
| |
| Jasen Betts 2006-01-18, 7:55 am |
| User-Agent: slrn/0.9.8.1 (Debian)
Message-ID: <3bac.43cdf18d.b33ad@clunker.homenet>
Date: Wed, 18 Jan 2006 07:43:09 -0000
Bytes: 390
X-Original-NNTP-Posting-Host: news.compass.net.nz
X-Original-Trace: 19 Jan 2006 00:13:02 +1300, news.compass.net.nz
Organization: CLEAR Net New Zealand http://www.clear.net.nz - Complaints abuse@clear.net.nz
Lines: 15
NNTP-Posting-Host: 203.97.37.6
X-Trace: sv3- s3JEYJlYkNeT88UUsjg1LgacGNqLIosofbm0gulu
rMqhO/ Uemn73rJvmqmnFu18XrcZbZ6HQwKB0jga!RSJQEY
ZsyCW+Ht6Ma8mqlmtxNPl0Lx35j/ 1Vwt7IJucET46hb1caXhomtkkZVvE4trQcGSXGNw
==
X-Complaints-To: Complaints to abuse@clear.net.nz
X-DMCA-Complaints-To: Complaints to abuse@clear.net.nz
X-Abuse-and-DMCA-Info: Please be sure to forward a copy of ALL headers
X-Abuse-and-DMCA-Info: Otherwise we will be unable to process your complaint properly
X-Postfilter: 1.3.32
Xref: number1.nntp.dca.giganews.com comp.compression:67885
On 2006-01-17, Chris Malek <chris.r.malek@gmail.com> wrote:
> Here is the XML message that I am really try to parse.
>
> The <data> element is really the deflated and encoded.
> <data encoding="base64(deflate)" encodedlength="808(605)" length="4950">eJzt
length isn't 808, I count 813
anyway your problem, could it be that the decoder needs to be primed.
Bye.
Jasen
| |
| Mark Adler 2006-01-18, 6:56 pm |
| Chris Malek wrote:
> Here is the XML message that I am really try to parse.
....
<data encoding="base64(deflate)" encodedlength="808(605)"
length="4950">
eJztmF1vmzAUhv9KlJteZdgQCEjElWNcghSc1Jiq
ubKiLpoqZd1Uomn79zMfTY1Jq1VJ14ty5/O+PseHk0dWILz8/X03+LV9LO5/ PEwv4BdwcYlClkvBMcsIT1ZC3ixzMqdcZmtGUHh1
v919FX9+botmH5M3ZM5lhMXgbrcpiumQD1FI09Ui
iQZ7tXE6JHPMh5YSb4Vc8lh2DFWo9AjmlPLjnqqP
JZs9uSxPZ7TyM3qtgo781PWxswinWFB5tcB
x11zxhFXH0cYrl6VXG7NciCXrptkskpxe5zQThtm
oMhJGQZVVLuUqny2SbG64ljFd1UBGcIr1KteS8wi
3McU0nMp8R5lAiJiTBbneGMypfz0iyWaqJqNuZoV
nxJaJbJhGUCM0I7E28eQ/ J8QbtjV4XZMqIMp9RwrPq51EJnKy2+RZv9BoXicf
NQbO72CtB/IA4BACAIAjcIrUbRsEPABtWGsXKf1TZ/KI95VI1f00wMETzsOEgNiqXVrDQKUQB9
1ZMmGCSidVVRV9o4ItW7O4JwBOqzNafFZjnRVtim
E7EqW1c0RFWyHhiIli146vwRtKsiLe+txCLK4tDS
BY1aVU0PTGrr3I6q46sqtCKTXwTKORmaCXE5S1PR
SK7P0KIDyi1q347w2HE9Fx5HGMLxGDi+/54Ig6MIO14wOQPCYDyy7R7hz8wv9GznA/idlFdnz2/P76n82q7vfsj96/f89vyezq8T2O/6F/jF+xf0/Pb8nsav+gGgM/Em//8VTt2
/ 53iFO53f9av81of4I9vrST4zydbhs4T1ykeyv+QR
mHM=
</data>
There are several lines of evidence that the quoted data is in error.
First, there are supposed to be 808 base64 characters in the data, but
there are 810. Furthermore, the single pad ("=") base64 character says
that there are supposed to be 4n+3 base64 characters before the pad,
but in fact there are 4n'+1, agreeing with the count mismatch.
Finally, zlib finds a compressed data error in the middle. One
hypothesis that matches the data is that two base64 characters were
inserted somewhere in the data (not necessarily next to each other),
which is causing the zlib detected error. The error is detected
somewhere around the "McU0", but the actual error is likely to be at
some point before that.
mark
|
|
|
|
|