Home > Archive > Visual Basic Syntax > May 2005 > Floating point run time error
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Floating point run time error
|
|
|
| Does anyone have an explanation for this:
Dim NewGamma As Single
Dim SingleTab() As Single
....
SingleTab(i) = SingleTab(i) ^ NewGamma
Kaboom!
'Invalid procedure call or argument'
?SingleTab(i)
-6.930858E-03
?NewGamma
0.4545454
?SingleTab(i) ^ NewGamma
Kaboom! Same as above.
?-6.930858E-03 ^ 0.4545454
-0.104361344064048
?-6.930858E-03 ^ NewGamma
-0.104361324197057
?SingleTab(i) ^ 0.4545454
Kaboom! Same as above.
And now just for fun:
SingleTab(i) = -0.104361344064048
?SingleTab(i)
-0.1043613
Hunky-dory. So, there's no overflow. (I tried doubles, anyway, and the
program still blows up.)
Any ideas what's going on and how do I fix it? My brain seems to be
stuck in reverse and I have an uneasy feeling some major
forehead-slapping is just around the corner... ;o)
Danny
(You guessed it! Remove NOSPAMFOR before emailing.)
| |
| Ken Halter 2005-05-13, 4:23 pm |
| "Danny" <NOSPAMFORdaniel_ahorn@yahoo.com> wrote in message
news:ufu481d9u0voej8goud6r4mb80l34psrlv@
4ax.com...
> ?SingleTab(i)
> -6.930858E-03
>
> ?NewGamma
> 0.4545454
>
> ?SingleTab(i) ^ NewGamma
>
> Kaboom! Same as above.
From Help....
The ^ Operator...
A number can be negative only if exponent is an integer value. When more
than one exponentiation is performed in a single expression, the ^ operator
is evaluated as it is encountered from left to right.
--
Ken Halter - MS-MVP-VB - http://www.vbsight.com
Sign up now to help keep VB support alive - http://classicvb.org/petition
Please keep all discussions in the groups..
| |
|
| Date: Wed, 11 May 2005 15:05:02 -0700
Name: "Ken Halter" <Ken_Halter@Use_Sparingly_Hotmail.com>
>"Danny" <NOSPAMFORdaniel_ahorn@yahoo.com> wrote in message
> news:ufu481d9u0voej8goud6r4mb80l34psrlv@
4ax.com...
>
>From Help....
>
>The ^ Operator...
>
>A number can be negative only if exponent is an integer value.
That still doesn't explain why it works when I type the number out as
a literal in direct mode. What I mean is this:
?SingleTab(i)
-6.930858E-03
?NewGamma
0.4545454
?-6.930858E-03 ^ 0.4545454
-0.104361344064048
is OK, as is:
?-6.930858E-03 ^ NewGamma
-0.104361324197057
Shouldn't both of them blow up as well? It seems to fail only when the
number is in a variable, which is exactly what I need in order to be
able to use it in a program.
Danny
(You guessed it! Remove NOSPAMFOR before emailing.)
| |
| Ken Halter 2005-05-13, 4:23 pm |
| "Danny" <NOSPAMFORdaniel_ahorn@yahoo.com> wrote in message
news:lqq681h2lstp2mmrd1sqg4pjsi68sc95lh@
4ax.com...
>
> That still doesn't explain why it works when I type the number out as
> a literal in direct mode. What I mean is this:
>
> ?SingleTab(i)
> -6.930858E-03
>
> ?NewGamma
> 0.4545454
>
> ?-6.930858E-03 ^ 0.4545454
> -0.104361344064048
>
> is OK, as is:
>
> ?-6.930858E-03 ^ NewGamma
> -0.104361324197057
>
> Shouldn't both of them blow up as well? It seems to fail only when the
> number is in a variable, which is exactly what I need in order to be
> able to use it in a program.
>
> Danny
>
Seems like they should blow up too... hmmm... here's a work-around if
interested....
'=========
Option Explicit
Private Sub Form_Load()
Dim SingleTab As Single
Dim NewGamma As Single
SingleTab = -0.006930858
NewGamma = 0.4545454
'Use Abs to make sure SingleTab's a positive number and then
'multiply the answer * the Sgn of SingleTab
Debug.Print (Abs(SingleTab) ^ NewGamma) * Sgn(SingleTab)
'Or, wrap it in a function...
Debug.Print MorePower(SingleTab, NewGamma)
End Sub
Private Function MorePower(NumA As Single, NumB As Single) As Double
MorePower = (Abs(NumA) ^ NumB) * Sgn(NumA)
End Function
'=========
--
Ken Halter - MS-MVP-VB - http://www.vbsight.com
DLL Hell problems? Try ComGuard - http://www.vbsight.com/ComGuard.htm
Sign up now to help keep VB support alive - http://classicvb.org/petition
Please keep all discussions in the groups..
| |
| Al Reid 2005-05-13, 4:23 pm |
| "Danny" <NOSPAMFORdaniel_ahorn@yahoo.com> wrote in message news:lqq681h2lstp2mmrd1sqg4pjsi68sc95lh@
4ax.com...
> Date: Wed, 11 May 2005 15:05:02 -0700
> Name: "Ken Halter" <Ken_Halter@Use_Sparingly_Hotmail.com>
>
>
> That still doesn't explain why it works when I type the number out as
> a literal in direct mode. What I mean is this:
>
> ?SingleTab(i)
> -6.930858E-03
>
> ?NewGamma
> 0.4545454
>
> ?-6.930858E-03 ^ 0.4545454
> -0.104361344064048
>
> is OK, as is:
>
> ?-6.930858E-03 ^ NewGamma
> -0.104361324197057
>
> Shouldn't both of them blow up as well? It seems to fail only when the
> number is in a variable, which is exactly what I need in order to be
> able to use it in a program.
>
It blows up if you do the following:
?csng(-6.930858E-03) ^ csng(0.4545454 )
and it also blows up if you do this:
?(-6.930858E-03) ^ (0.4545454)
so it seems that the immediate window is evaluating it as
?-(6.930858E-03 ^ 0.4545454)
and therefore it does not blow up.
This would indicate that both Ken and the documentation are correct.
--
Al Reid
> Danny
>
> (You guessed it! Remove NOSPAMFOR before emailing.)
| |
|
| Date: Thu, 12 May 2005 08:49:25 -0700
Name: "Ken Halter" <Ken_Halter@Use_Sparingly_Hotmail.com>
>Seems like they should blow up too... hmmm...
Hmmm, is right! ;o) That inconsistency is what caused my head
scratching...
>here's a work-around if interested....
Very much so! Thank you!
Actually, I already implemented a similar workaround only without a
function to make it run a bit faster.
The trouble is I wanted to make this efficient (we're talking 100 MB+
graphics files here) and all these extra checks and wrapping don't
help.
What's so annoying is that if they could make it work in immediate
mode why the (wanton?) limitation at run time!? The mystery of MS...
:-/
Danny
(You guessed it! Remove NOSPAMFOR before emailing.)
| |
|
| Date: Thu, 12 May 2005 11:51:45 -0400
Name: "Al Reid" <areidjr@reidDASHhome.com>
>It blows up if you do the following:
>
>?csng(-6.930858E-03) ^ csng(0.4545454 )
>
>and it also blows up if you do this:
>
>?(-6.930858E-03) ^ (0.4545454)
>
>so it seems that the immediate window is evaluating it as
>
>?-(6.930858E-03 ^ 0.4545454)
>
>and therefore it does not blow up.
>
>This would indicate that both Ken and the documentation are correct.
Well, with my pedant hat firmly on, the documentation is not strictly
correct. It should say it doesn't work only if the (negative) number
is in a variable. It works just fine if the number is a literal.
However, that still doesn't explain why the immediate mode evaluation
was not implemented at run time? Or, more importantly, why are the two
inconsistent!?
Indeed, if it works in immediate mode without any problems, it only
begs the question why the (seemingly unnecessary) limitation at run
time?
Danny
(You guessed it! Remove NOSPAMFOR before emailing.)
| |
| Al Reid 2005-05-13, 4:23 pm |
| "Danny" <NOSPAMFORdaniel_ahorn@yahoo.com> wrote in message news:pr3781hdogrklspamdltccosic0daf3uqt@
4ax.com...
> Date: Thu, 12 May 2005 11:51:45 -0400
> Name: "Al Reid" <areidjr@reidDASHhome.com>
>
>
> Well, with my pedant hat firmly on, the documentation is not strictly
> correct. It should say it doesn't work only if the (negative) number
> is in a variable. It works just fine if the number is a literal.
>
But it doesn't work if it's a literal as I demonstrated above:
?(-6.930858E-03) ^ (0.4545454) < < this throws the error because number is negative and exponent is not an integer
where as
?-6.930858E-03 ^ 0.4545454
is evaluated as
?-(6.930858E-03 ^ 0.4545454)
> However, that still doesn't explain why the immediate mode evaluation
> was not implemented at run time? Or, more importantly, why are the two
> inconsistent!?
>
As I stated above, it's due to the oder in which the operators are evaluated.
> Indeed, if it works in immediate mode without any problems, it only
> begs the question why the (seemingly unnecessary) limitation at run
> time?
>
> Danny
>
> (You guessed it! Remove NOSPAMFOR before emailing.)
--
Al Reid
| |
| Russ Holsclaw 2005-05-13, 4:23 pm |
| > Well, with my pedant hat firmly on, the documentation is not strictly
> correct. It should say it doesn't work only if the (negative) number
> is in a variable. It works just fine if the number is a literal.
>
Well, as others have pointed out, the apparant paradox is actually due to
the fact that the exponentiation operator ('^') takes precedence over the
unary negation operator (the '-' in front of the first operand). The left
operand is a positive number, because of the order of evaluation of the
expression. Your error is in thinking that the minus-sign would be treated
as part of the literal.
So, you weren't quite pedantic *enough* for this one. Looks like you need
to tighten the chin-strap on your hat. ;-)
Now, if this were Fortran, and parentheses were placed to ensure proper
evaluation, I think you'd get a Complex result -- that is, a number with
both Real and Imaginary components. ... but it's not.
| |
|
| Date: Thu, 12 May 2005 12:47:52 -0600
Name: "Russ Holsclaw" <russ@holsclaw.nyet>
>So, you weren't quite pedantic *enough* for this one. Looks like you need
>to tighten the chin-strap on your hat. ;-)
Indeed, the hat was not on as firmly as I thought! :-)
Nevertheless, balancing my flimsy headgear, I can still blame MS for
the inconsistency - as pedants with hurt pride would do, rather than
admit a mistake... ;o)
Danny
(You guessed it! Remove NOSPAMFOR before emailing.)
| |
|
| Date: Thu, 12 May 2005 13:28:52 -0400
Name: "Al Reid" <areidjr@reidDASHhome.com>
>But it doesn't work if it's a literal as I demonstrated above:
>
>?(-6.930858E-03) ^ (0.4545454) < < this throws the error because number is negative and exponent is not an integer
>
>where as
>
>?-6.930858E-03 ^ 0.4545454
>
>is evaluated as
>
>?-(6.930858E-03 ^ 0.4545454)
I stand corrected, demonstrating once again that pedantry is a
double-edged sword not to be used lightly! ;o)
But - paper cuts notwithstanding - there is still an inconsistency
between immediate and run time in how the expressions are evaluated,
and that's definitely wrong i.e., the "bug" was elsewhere.
However, be that as it may, you were correct.
Danny
(You guessed it! Remove NOSPAMFOR before emailing.)
| |
| Tony Proctor 2005-05-13, 4:23 pm |
| The exponentiation operator is evaluated before the unary negation operator.
Hence,
-6.930858E-03 ^ 0.4545454
is evaluated as -(6.930858E-03 ^ 0.4545454) rather than (-6.930858E-03) ^
0.4545454
In the case of your run-time version, though, the -ve state of the first
operand is implicit in its stored value. There is no operator precedence
issue. Hence, the result is different
Tony Proctor
"Danny" <NOSPAMFORdaniel_ahorn@yahoo.com> wrote in message
news:dr3781p62v9k18i7cibag5soo2trf9qroq@
4ax.com...
> Date: Thu, 12 May 2005 08:49:25 -0700
> Name: "Ken Halter" <Ken_Halter@Use_Sparingly_Hotmail.com>
>
>
> Hmmm, is right! ;o) That inconsistency is what caused my head
> scratching...
>
>
> Very much so! Thank you!
>
> Actually, I already implemented a similar workaround only without a
> function to make it run a bit faster.
>
> The trouble is I wanted to make this efficient (we're talking 100 MB+
> graphics files here) and all these extra checks and wrapping don't
> help.
>
> What's so annoying is that if they could make it work in immediate
> mode why the (wanton?) limitation at run time!? The mystery of MS...
> :-/
>
> Danny
>
> (You guessed it! Remove NOSPAMFOR before emailing.)
| |
| Al Reid 2005-05-13, 4:23 pm |
| "Danny" <NOSPAMFORdaniel_ahorn@yahoo.com> wrote in message news:9ve7819325rt4d0pe5a8njpol3oskpg47g@
4ax.com...
> Date: Thu, 12 May 2005 13:28:52 -0400
> Name: "Al Reid" <areidjr@reidDASHhome.com>
>
>
> I stand corrected, demonstrating once again that pedantry is a
> double-edged sword not to be used lightly! ;o)
>
> But - paper cuts notwithstanding - there is still an inconsistency
> between immediate and run time in how the expressions are evaluated,
> and that's definitely wrong i.e., the "bug" was elsewhere.
>
> However, be that as it may, you were correct.
>
> Danny
>
> (You guessed it! Remove NOSPAMFOR before emailing.)
Another thing to keep in mind as you look for workarounds for this apparent
"bug" in the ^ operator is that the correct answer is not
-0.104361344064048. Since VB and most scientific calculators do not handle
complex numbers, they display or throw an error when the result would yield
a complex number.
--
Al Reid
| |
| Ken Halter 2005-05-13, 4:23 pm |
| "Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in message
news:%23MIhTe6VFHA.2128@TK2MSFTNGP14.phx.gbl...
> The exponentiation operator is evaluated before the unary negation
> operator.
> Hence,
>
Yeah... I had a friend with a unary infection... he said it hurt like heck
to go to the bathroom (I know.... but it's Friday and all) ;-)
--
Ken Halter - MS-MVP-VB - http://www.vbsight.com
DLL Hell problems? Try ComGuard - http://www.vbsight.com/ComGuard.htm
Sign up now to help keep VB support alive - http://classicvb.org/petition
Please keep all discussions in the groups..
| |
|
| Date: Fri, 13 May 2005 07:12:43 -0700
Name: "Ken Halter" <Ken_Halter@Use_Sparingly_Hotmail.com>
>Yeah... I had a friend with a unary infection... he said it hurt like heck
>to go to the bathroom (I know.... but it's Friday and all) ;-)
ROTFL! And, don't forget, it's Friday the *13th*! So, today is the day
to do all those critical modifications! ;o)
Danny
(You guessed it! Remove NOSPAMFOR before emailing.)
| |
|
| Date: Fri, 13 May 2005 08:13:18 -0400
Name: "Al Reid" <areidjr@reidDASHhome.com>
>Another thing to keep in mind as you look for workarounds for this apparent
>"bug" in the ^ operator is that the correct answer is not
>-0.104361344064048. Since VB and most scientific calculators do not handle
>complex numbers, they display or throw an error when the result would yield
>a complex number.
Yes, I sort of had that in the back of my mind. Not all floating point
numbers can be accurately expressed outside of floating point
notation.
However, the data I'm massaging is pixels in a floating point i.e.
high dynamic range (HDR) image. After I apply gamma I convert it to
16-bit so there's a lot of truncation and other inaccuracies going on
there, much worse than the above.
I could just as well convert first and apply gamma to 16-bit pixel
values but I found I get better results if I do that while the image
is still in floating point domain. But the improvements are really
theoretical because they are not visible on a garden-variety 8-bit
displays not to mention our < 8-bit eyes...
BTW, one of the tests I did ("just for fun") was to convert 16-bit
values back to floating point representation just to see how much
"damage" there is and for this particular application (image pixels)
it wasn't really that bad. Of course, for a pixel value a full blow
32-bit floating point is an overkill, anyway, as most of it is noise
which is why regular HDR formats don't even store the full floating
point value.
Danny
(You guessed it! Remove NOSPAMFOR before emailing.)
| |
|
| Yes, I think the confusion/inconsistency is due to how the "-" symbol
is interpreted.
In immediate mode, because "-" is stated explicitly, it is considered
an operator.
In run-time mode, as you say, it's implicit and therefore treated as a
sign.
So, in the end, and grudgingly ;o) I guess MS is off the hook as the
problem is due to the fact that we use the same symbol "-" to mean two
different things.
Date: Fri, 13 May 2005 11:36:06 +0100
Name: "Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com>
>The exponentiation operator is evaluated before the unary negation operator.
>Hence,
>
> -6.930858E-03 ^ 0.4545454
>
>is evaluated as -(6.930858E-03 ^ 0.4545454) rather than (-6.930858E-03) ^
>0.4545454
>
>In the case of your run-time version, though, the -ve state of the first
>operand is implicit in its stored value. There is no operator precedence
>issue. Hence, the result is different
>
> Tony Proctor
Danny
(You guessed it! Remove NOSPAMFOR before emailing.)
|
|
|
|
|