Home > Archive > ASP > June 2007 > <%=1/15%> retuns 6.66666666666667E-02
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
<%=1/15%> retuns 6.66666666666667E-02
|
|
|
| yes <%=1/15%> retuns 6.66666666666667E-02
anyone knows why or where i am wrong?
| |
| Bob Barrows [MVP] 2007-06-25, 6:56 pm |
| ..nLL wrote:
> yes <%=1/15%> retuns 6.66666666666667E-02
>
> anyone knows why or where i am wrong?
Take a look at this series of blog articles about floating point
arithmetic. It starts here:
http://blogs.msdn.com/ericlippert/a.../10/350108.aspx
You can find the rest of the series using the link to the chronological
list of posts on the right side of the page.
--
Microsoft MVP -- ASP/ASP.NET
Please reply to the newsgroup. The email account listed in my From
header is my spam trap, so I don't check it very often. You will get a
quicker response by posting to the newsgroup.
| |
|
| This is a bit complicated than i expected. is there any function that
will help me to return proper result (like 0.06) from 1/15?
Thank you
On Jun 25, 1:48 pm, "Bob Barrows [MVP]" <reb01...@NOyahoo.SPAMcom>
wrote:
> .nLL wrote:
>
>
> Take a look at this series of blog articles about floating point
> arithmetic. It starts here:http://blogs.msdn.com/ericlippert/a.../10/350108.aspx
>
> You can find the rest of the series using the link to the chronological
> list of posts on the right side of the page.
>
> --
> Microsoft MVP -- ASP/ASP.NET
> Please reply to the newsgroup. The email account listed in my From
> header is my spam trap, so I don't check it very often. You will get a
> quicker response by posting to the newsgroup.
| |
| Bob Barrows [MVP] 2007-06-25, 6:56 pm |
| OK, a simple answer:
No.
The correctly rounded result is 0.07, so if you want the result to be
"0.06", you will need to write complicated code to round the result the
way you want to (why do you want to round down?).
There is no substitute to learning how to properly deal with f-p
arithmetic. This will not be the last time you encounter this.
For a simple answer, try the vbscript FormatNumber function.
..nLL wrote:[color=darkred]
> This is a bit complicated than i expected. is there any function that
> will help me to return proper result (like 0.06) from 1/15?
>
> Thank you
>
>
> On Jun 25, 1:48 pm, "Bob Barrows [MVP]" <reb01...@NOyahoo.SPAMcom>
> wrote:
--
Microsoft MVP -- ASP/ASP.NET
Please reply to the newsgroup. The email account listed in my From
header is my spam trap, so I don't check it very often. You will get a
quicker response by posting to the newsgroup.
| |
|
| thank you formatnumber was the answer i was looking for
On Jun 25, 2:41 pm, "Bob Barrows [MVP]" <reb01...@NOyahoo.SPAMcom>
wrote:
> OK, a simple answer:
> No.
>
> The correctly rounded result is 0.07, so if you want the result to be
> "0.06", you will need to write complicated code to round the result the
> way you want to (why do you want to round down?).
>
> There is no substitute to learning how to properly deal with f-p
> arithmetic. This will not be the last time you encounter this.
>
> For a simple answer, try the vbscript FormatNumber function.
>
>
>
> .nLL wrote:
>
>
>
>
>
>
>
> --
> Microsoft MVP -- ASP/ASP.NET
> Please reply to the newsgroup. The email account listed in my From
> header is my spam trap, so I don't check it very often. You will get a
> quicker response by posting to the newsgroup.
|
|
|
|
|