Compression - QI please don't laugh at it.

Author

QI please don't laugh at it.

David A. Scott

2006-02-08, 8:54 am

People here we do try to help new commers and its clear nightlight
needs a little help in the concept of compression.

I read the last couple of posts from nightlight I tried not
to laugh. Could someone please explain what nitelite.zip
shows. He still does not even understand basic compression.

Yes NIGHTLIGHT the plzip does compress the test files hundreds
of time smaller than my code. But compression is a trade off you
may not understand the counting theorm but its real.

The goal was to compress a file made of 3 sperate symbols each which
has a 1/3 chance of occuring and is not dependent on what occured before.

This may be new to you but look at the shorter file
AAAAAAAAAA 10 A's and the file
AACBABBCCB 3 A's 4 B's and 3 C's

If one allowed 10 symbols to occur and then stopped the file
of 10 A's it is as likely as the second file. Yes to the eye they
appear different but each is exactly as likely. I included the
extreme test cases to show that it works. Anyone can pick other
files. I am sure you have people where you work that could produce
such files for you if the task is to hard. If you think they were
cherry picked and the rest would fail you need to check again
the code is not that hard to understand surely you have some
people who work for you who could verify that in a matter of
minutes. But like I said in last post I am not god and if one
finds an error than I learn. That is why its best to code to
files so any one can check unless one wishs to hide errors so
others don't have a chance to check it. The down side of not
useing files is that if you don't have errors pointed out to you
then you would fail to learn but thats up to you.

The counting theorm kind of works like this. Compression is
nothing but an ordering of the possible output files at best.
If my code compressed the all A's to what pkzip did some other
file would have to be mapped ( compressed ) to a larger size.
There is no free lunch.

If one took all equially likely files at the length of the file
of the A's pkzip would on the average compress worse than my code.
But that is to be expected since pkzip is designed to work on
many file types. In fact most files would compress to a larger size.
Since most would appear random to pkzip and because pkzip does
not fully use the output space.

What you still seem unable to acknowlege is that a bijective
arithmetic file compressor has no gaps. That means every possible
binary file would decompress uniquely to a file of the type the
compressor was designed to work on. So no file compression method
would work better as files get longer. You could beat it sometime
and not beat it other times. But if you can't compress bijectively
to the file space then in the long run your QI would be the loser
for these types of file sources.

Another minor point you glossed over is that QI gets the index
its not hard to write an airhtmetic that gets the index. Which means
why would one use QI anyway especially for long files if its easier
to get the working index with arihtmetic in the first place.

Is it possible to get you to give an anwser without quoting volumes
of reference. Its real code that people here are interested in. I have
had many people who lack a real understanding of arihtmeic file
compression quote texts that say it can't be done. Some quotes are
plain wrong other quotes are valid but the person quoting them really
doesn't understand them, I feel like your a preacher who is unsure
about god arguing with another preacher about god. Both of who are
not listening to each other but each trying to show off by quoting
voluminous verses in the bible. Here we like to see plain facts and
working code. Not repeated stuff about how much better QI will make
BTW compression work. We see enough pie in the sky. just SHOW ME I
lived in MO for a long time.

David A. Scott
--
My Crypto code
http://bijective.dogma.net/crypto/scott19u.zip
http://www.jim.com/jamesd/Kong/scott19u.zip old version
My Compression code http://bijective.dogma.net/
**TO EMAIL ME drop the roman "five" **
Disclaimer:I am in no way responsible for any of the statements
made in the above text. For all I know I might be drugged.
As a famous person once said "any cryptograhic
system is only as strong as its weakest link"

David A. Scott

2006-02-08, 8:54 am

"David A. Scott" <daVvid_a_scott@email.com> wrote in
news:Xns975F89659A26BH110W296LC45WIN3030
R@213.155.197.138:

> People here we do try to help new commers and its clear nightlight
> needs a little help in the concept of compression.
>
> I read the last couple of posts from nightlight I tried not
> to laugh. Could someone please explain what nitelite.zip
> shows. He still does not even understand basic compression.
>
>

Sorry here it is in case you can't find it
http://bijective.dogma.net/nitelite.zip

David A. Scott
--
My Crypto code
http://bijective.dogma.net/crypto/scott19u.zip
http://www.jim.com/jamesd/Kong/scott19u.zip old version
My Compression code http://bijective.dogma.net/
**TO EMAIL ME drop the roman "five" **
Disclaimer:I am in no way responsible for any of the statements
made in the above text. For all I know I might be drugged.
As a famous person once said "any cryptograhic
system is only as strong as its weakest link"

Earl Colby Pottinger

2006-02-08, 8:54 am

"David A. Scott" <daVvid_a_scott@email.com> :

> People here we do try to help new commers and its clear nightlight
> needs a little help in the concept of compression.
>
> I read the last couple of posts from nightlight I tried not
> to laugh. Could someone please explain what nitelite.zip
> shows. He still does not even understand basic compression.

The problem is not that he does not understand but rather he does not want to
understand.

The facts are simple:

NightLight believes his code QI is better than AC.
People tested his supplied code against AC.
QI fails.
NightLight can not accept his code is the best. Thus refuses to follow the
logic other supply.

At present NightLight does not seem to be a kook, kooks rarely (very rarely)
supply working code.

However, he appears to be showing the first signs of kookdom, refusal to look
at real world numbers supplied people doing thier own experiments and
resistance to changing his own code for testing.

Earl Colby Pottinger

--
<Cruising, building a Catamaran, Rebuilding Cabin, New Peroxide Still Design,
Writting SF, Programming FOSS - What happened to the time?>

Sachin Garg

2006-02-08, 8:54 am

Earl Colby Pottinger wrote:
> "David A. Scott" <daVvid_a_scott@email.com> :
>
>
> The problem is not that he does not understand but rather he does not want to
> understand.
>
> The facts are simple:
>
> NightLight believes his code QI is better than AC.
> People tested his supplied code against AC.
> QI fails.
> NightLight can not accept his code is the best. Thus refuses to follow the
> logic other supply.
>
> At present NightLight does not seem to be a kook, kooks rarely (very rarely)
> supply working code.
>
> However, he appears to be showing the first signs of kookdom, refusal to look
> at real world numbers supplied people doing thier own experiments and
> resistance to changing his own code for testing.

QI seems to be something interesting and is real for sure, we have seen
code.

I believe parts of his claims are correct, but he should stop saying
broad terms like "always" better than "everything" else on "both" speed
and ratio. He should stick only to claims which he can prove with code
(and keep his posts short and to-the-point).

Another issue with "acceptance" of QI has been that it requires a
change in how we think about modelling. I don't know if this modelling
scheme is better than the old one, but even if it is actually better,
this will take time to be absorbed by everyone. He shouldn't expect it
to happen overnight.

Also, the fact that QI is patented, IMHO, does reduce the volunteers
who will want to work on it or with it. My guess is that he will
himself have to do the efforts.

I do not agree with the 'kookdom' part. Even if it is not as useful as
he says, QI is real and it would have taken effort to conceptualize it.

Sachin Garg [India]
http://www.sachingarg.com

Sachin Garg

2006-02-08, 8:54 am

Sachin Garg wrote:
> Earl Colby Pottinger wrote:
>
> QI seems to be something interesting and is real for sure, we have seen
> code.
>
> I believe parts of his claims are correct, but he should stop saying
> broad terms like "always" better than "everything" else on "both" speed
> and ratio. He should stick only to claims which he can prove with code
> (and keep his posts short and to-the-point).
>
> Another issue with "acceptance" of QI has been that it requires a
> change in how we think about modelling. I don't know if this modelling
> scheme is better than the old one, but even if it is actually better,
> this will take time to be absorbed by everyone. He shouldn't expect it
> to happen overnight.
>
> Also, the fact that QI is patented, IMHO, does reduce the volunteers
> who will want to work on it or with it. My guess is that he will
> himself have to do the efforts.
>
> I do not agree with the 'kookdom' part. Even if it is not as useful as
> he says, QI is real and it would have taken effort to conceptualize it.

Some of the questions related to QI which are currently being discussed
and debated were answered by nightlight in a mail exchange when this
whole thing started in November.

Fortunately the answers were brief and understandable. You can read
them at http://www.c10n.info/archives/251

For other QI related stories:
http://www.c10n.info/index.php?s=quantized+indexing

I hope this helps.

Sachin Garg [India]
http://www.sachingarg.com

Phil Carmody

2006-02-08, 8:54 am

Earl Colby Pottinger <earlcolby.pottinger@sympatico.ca> writes:

> "David A. Scott" <daVvid_a_scott@email.com> :
>
>
> The problem is not that he does not understand but rather he does not want to
> understand.
>
> The facts are simple:
>
> NightLight believes his code QI is better than AC.
> People tested his supplied code against AC.
> QI fails.
> NightLight can not accept his code is the best. Thus refuses to follow the
^not?
> logic other supply.

I disagree with those "facts".

NL is just what we'd call here a comma-XXXXer. He's obsessive about
one particular aspect, which apparently (I've not run his code, but
I don't disbelieve claims about its output) he deems more important
than anything else, despite the fact that the difference between his
results and others in his chosen metric are exceptionally small.
(Hundredths of a %, if I remember David's numbers). If other metrics
are used to compare codecs, ones which include speed and range of
potential use, then his algorithm loses that cigarette-paper thin
edge that it previously had.

He just doesn't have, or want, the "bigger picture".

The fact that he was utterly ignorant of the fact that one can perform
upwards of 200 multiplications in the time it takes to make one main
memory access on some modern architectures, and the fact that he's
utterly ignorant of the fact that caches typically take up far more die
area than multiplier units on such architectures, also indicates that
there are other aspects he's claimed to have knowledge about, yet been
found wanting.

> At present NightLight does not seem to be a kook, kooks rarely (very rarely)
> supply working code.
>
> However, he appears to be showing the first signs of kookdom, refusal to look
> at real world numbers supplied people doing thier own experiments and
> resistance to changing his own code for testing.

It's more like setting up self-chosen self-serving artificial criteria
such that other people's numbers are irrelevant. Which is similar in
its heading-towards-kookdom behaviour. I'd lay the blame in part at
good old-fashioned ivory-tower academic behaviour, with the associated
total inability to communicate with others who don't already think the
same way.

Phil
--
What is it: is man only a blunder of God, or God only a blunder of man?
-- Friedrich Nietzsche (1844-1900), The Twilight of the Gods

Phil Carmody

2006-02-08, 8:54 am

"Sachin Garg" <schngrg@gmail.com> writes:
> Some of the questions related to QI which are currently being discussed
> and debated were answered by nightlight in a mail exchange when this
> whole thing started in November.
>
> Fortunately the answers were brief and understandable. You can read
> them at http://www.c10n.info/archives/251

Needs correcting:
<<<
Ratko V. Tomic, Chief scientist & CTO, 1stWorks Corp. has been talking about a new algorithm for entropy coding.[color=darkred]

It's not "new" at all. I remember looking at on something extremely
close to that with Roger Fawcett of Oxford University's robotics lab,
as he was working on it as part of his doctoral thesis, way back in
the 90s.

Phil
--
What is it: is man only a blunder of God, or God only a blunder of man?
-- Friedrich Nietzsche (1844-1900), The Twilight of the Gods

Sachin Garg

2006-02-08, 8:54 am

Phil Carmody wrote:
> "Sachin Garg" <schngrg@gmail.com> writes:
>
> Needs correcting:
> <<<
> Ratko V. Tomic, Chief scientist & CTO, 1stWorks Corp. has been talking about a new algorithm for entropy coding.
>
> It's not "new" at all. I remember looking at on something extremely
> close to that with Roger Fawcett of Oxford University's robotics lab,
> as he was working on it as part of his doctoral thesis, way back in
> the 90s.

Thanks for pointing that out. I searched a bit and found this:

http://www.informatik.uni-trier.de/...cett:Roger.html
http://wotan.liu.edu/docis/dbl/dccd..._434_CCANEC.htm

But too few details. I wonder if someone can provide more information
on this.

Sachin Garg [India]
http://www.sachingarg.com

Earl Colby Pottinger

2006-02-08, 8:54 am

Earl Colby Pottinger <earlcolby.pottinger@sympatico.ca>:

> NightLight believes his code QI is better than AC.
> People tested his supplied code against AC.
> QI fails to beat AC in either compression ratio or speed.
> NightLight can not accept his code is not the best. Thus refuses to follow
> the logic other supply.

Phil Carmody <thefatphil_demunged@yahoo.co.uk> :

> I disagree with those "facts".

I am not sure why you do. Yes I was missing a 'not' - dumb of me. I revised
my statements a little.

Maybe I did not state all the things wrong with NightLight's appoach to
trying to get people to acept his compression method but I don't believe I am
that far off.

However, I can honestly say that I have read less that 30% of what he had
wrote as even the simplest request on why a part of his code is written in
certain way is answered in reams of quotes from other reseach that seems to
have little to do with the question.

> NL is just what we'd call here a comma-XXXXer. He's obsessive about
> one particular aspect, which apparently (I've not run his code, but
> I don't disbelieve claims about its output) he deems more important
> than anything else, despite the fact that the difference between his
> results and others in his chosen metric are exceptionally small.

I must of missed that part (or misread it) it was my understanding from
reading the messages that other codecs were compressing by a large precentage
more.

> (Hundredths of a %, if I remember David's numbers). If other metrics
> are used to compare codecs, ones which include speed and range of
> potential use, then his algorithm loses that cigarette-paper thin
> edge that it previously had.

Sorry, where was his code doing that well. Is this about the side-channels
(a part I have not followed well)

> He just doesn't have, or want, the "bigger picture".

That I have to agree to considering at one time he did not see the use of
testing files, including the use of the same files to test diffirent codecs.

> The fact that he was utterly ignorant of the fact that one can perform
> upwards of 200 multiplications in the time it takes to make one main
> memory access on some modern architectures, and the fact that he's
> utterly ignorant of the fact that caches typically take up far more die
> area than multiplier units on such architectures, also indicates that
> there are other aspects he's claimed to have knowledge about, yet been
> found wanting.

Ok, I agree that is 'ivory-tower academic behaviour' if you want your code to
working in the real world. There is nothing wrong not knowing this info if
you are just working out the model of a new compression method, but I believe
that NightLight claims of being better than AC is based on his model of what
CPU resources are used in QI vs AC. If his model is wrong, then his
conculsion most likely will be too. Did I get that right from your point of
view?

[color=darkred]
[color=darkred]
> It's more like setting up self-chosen self-serving artificial criteria
> such that other people's numbers are irrelevant.

I agree, but it is a criteria in which real tests still has him loses out to
other codecs.

> Which is similar in
> its heading-towards-kookdom behaviour. I'd lay the blame in part at
> good old-fashioned ivory-tower academic behaviour, with the associated
> total inability to communicate with others who don't already think the
> same way.

Do you mean that is his personal behaviour or that he is in a
university/institution that supports his actions too?

Earl Colby Pottinger

PS. If he will not look at experimental results from others how can you get
such a person to change thier mind.

--
<Cruising, building a Catamaran, Rebuilding Cabin, New Peroxide Still Design,
Writting SF, Programming FOSS - What happened to the time?>

Earl Colby Pottinger

2006-02-08, 8:54 am

"Sachin Garg" <schngrg@gmail.com> :

> I do not agree with the 'kookdom' part. Even if it is not as useful as
> he says, QI is real and it would have taken effort to conceptualize it.

I did not say he was a kook, I said he was in danger of becoming one. Look
at the first posts of JC years ago for example.

As far as I can tell like many others he believed he came up with a new idea
at looking at compression. Like many other he was asked to show code. First
his code was not finish, and later he only gave snippets. But the most
important part was his code in the long run did not work. He could not meet
his own expectations.

At this point a normal coder pushs the code aside and goes back to thier
model to see where they went wrong. Then they either dump the model as
wrong, revise the model and start over or realize thier code did not match
the model and start recoding.

JC struck to the same model and code and has kept pushing the code to more
and more complex levels without ever producing something he can shows works.
Lately he has made the move from kook to con.

This where NightLife seems to start entering kookdom, not because his method
is new(?) or odd. But because he will not revise his model or code on the
basis of real world experiments done by others.

Earl Colby Pottinger

PS. Flame me if I am wrong, I admit to not reading all of NightLife
postings, there is just too much unneed wordage.

--
<Cruising, building a Catamaran, Rebuilding Cabin, New Peroxide Still Design,
Writting SF, Programming FOSS - What happened to the time?>

Sachin Garg

2006-02-08, 8:54 am

Earl Colby Pottinger wrote:
> "Sachin Garg" <schngrg@gmail.com> :
>
>
> I did not say he was a kook, I said he was in danger of becoming one. Look
> at the first posts of JC years ago for example.
>
> As far as I can tell like many others he believed he came up with a new idea
> at looking at compression. Like many other he was asked to show code. First
> his code was not finish, and later he only gave snippets. But the most
> important part was his code in the long run did not work. He could not meet
> his own expectations.
>
> At this point a normal coder pushs the code aside and goes back to thier
> model to see where they went wrong. Then they either dump the model as
> wrong, revise the model and start over or realize thier code did not match
> the model and start recoding.
>
> JC struck to the same model and code and has kept pushing the code to more
> and more complex levels without ever producing something he can shows works.
> Lately he has made the move from kook to con.
>
> This where NightLife seems to start entering kookdom, not because his method
> is new(?) or odd. But because he will not revise his model or code on the
> basis of real world experiments done by others.
>
> Earl Colby Pottinger
>
> PS. Flame me if I am wrong, I admit to not reading all of NightLife
> postings, there is just too much unneed wordage.

I have to admit I didn't read all his posts either. Its not very
motivating to read details about something about which its author can
find all the time to do long usenet posts but not some time to code a
practically useful application.

I just hope that these 'kook' and 'con' predictions are not true.

Sachin Garg [India]
http://www.sachingarg.com

Earl Colby Pottinger

2006-02-08, 8:54 am

"Sachin Garg" <schngrg@gmail.com> :

> I just hope that these 'kook' and 'con' predictions are not true.

Maybe, I should not have mentioned JC's con attempts. Sorry there. A lot of
people with kooky ideas about compression have come and gone in this UseNet
group over the years, very few have moved on to out and out cons.

And remember I am not saying he is one, just that he may become one - only
time will tell.

Earl Colby Pottinger

--
<Cruising, building a Catamaran, Rebuilding Cabin, New Peroxide Still Design,
Writting SF, Programming FOSS - What happened to the time?>

Phil Carmody

2006-02-08, 8:54 am

Earl Colby Pottinger <earlcolby.pottinger@sympatico.ca> writes:
> Maybe I did not state all the things wrong with NightLight's appoach to
> trying to get people to acept his compression method but I don't believe I am
> that far off.

I do not believe you are that far off either.
I suspect I'm more sympathetic to NL than you, I just _really_
dislike his presentation.

Phil
--
What is it: is man only a blunder of God, or God only a blunder of man?
-- Friedrich Nietzsche (1844-1900), The Twilight of the Gods

Sponsored Links