Wednesday, March 7, 2012

help with percision? if you enter a number in the trillions such 9,999,999,999,999 .net or s

can you please explian this chart:

The operand expressions are denoted as expression e1, with precision p1 and scale s1, and expression e2, with precision p2 and scale s2. The precision and scale for any expression that is not decimal is the precision and scale defined for the data type of the expression.

Operation

Result precision

Result scale *

e1 + e2

max(s1, s2) + max(p1-s1, p2-s2) + 1

max(s1, s2)

e1 - e2

max(s1, s2) + max(p1-s1, p2-s2) + 1

max(s1, s2)

e1 * e2

p1 + p2 + 1

s1 + s2

e1 / e2

p1 - s1 + s2 + max(6, s1 + p2 + 1)

max(6, s1 + p2 + 1)

e1 { UNION | EXCEPT | INTERSECT } e2

max(s1, s2) + max(p1-s1, p2-s2)

max(s1, s2)

* The result precision and scale have an absolute maximum of 38. When a result precision is greater than 38, the corresponding scale is reduced to prevent the integral part of a result from being truncated.

e1 = numeric(20,8)

e2 = numeric(20,8)

e1/e2

is this correct

max(6,s1 + p2 + 1)

8 + 20 + 1= 29 since under 38 use 29 scale

scale would be max(6,29) = 29 correct?

p1 - s1 + s2 + max(6, s1 + p2 + 1)

20 - 8 + 8 + 29 = 49 does that mean it truncate the least sugificant digits by 29 - 11 = 18 so the effective result should be numeric(38,18) or ##,###,###,###,###,###,###.000000000000000000 this does not seem to be what you get can some explain also we have seen that if you enter a number in the trillions such 9,999,999,999,999 neither .net or sql management studio cannot display the value?

I think you have it, though it turns out that the actual returned type is numeric(38,17). You can see this using a variant:


declare @.c sql_variant

declare @.a numeric(20,8) --set the datatypes here
declare @.b decimal(20,8) --set the datatypes here

set @.a = 1 --set a value here
set @.b = 1 --set a value here

select @.c = @.a / @.b --do the math
select cast(@.c as varchar(40)),
cast(sql_variant_property(@.c,'BaseType') as varchar(20)) + '(' +
cast(sql_variant_property(@.c,'Precision') as varchar(10)) + ',' +
cast(sql_variant_property(@.c,'Scale') as varchar(10)) + ')'


- -
1.000000000000000000 numeric(38,18)

Don't quite understand your issue with trillions (though you have to use 21,8 instead of 20, 8 for the datatype:


declare @.c sql_variant

declare @.a numeric(21,8) --set the datatypes here

set @.a = 9999999999999 --set a value here

select @.c = @.a
select cast(@.c as varchar(40)),
cast(sql_variant_property(@.c,'BaseType') as varchar(20)) + '(' +
cast(sql_variant_property(@.c,'Precision') as varchar(10)) + ',' +
cast(sql_variant_property(@.c,'Scale') as varchar(10)) + ')'

- -
9999999999999.00000000 numeric(21,8)

No comments:

Post a Comment