PowerShell Weekly
PowerShell Weekly

PoshBytes: Numeric Precision in PowerShell

Matthew Dowst

PoshBytes: Numeric Precision in PowerShell

In this PoshBytes short you will see how PowerShell can silently lose precision when big integer math is promoted to double and how using the right numeric types and decimal literals helps keep your results exact.

This post is a companion for the video embedded below. Scroll down to see the code from the video.

Tiny numbers, tiny problems

Small integers stay as Int32 and are exact

2 + 2
(2 + 2).GetType().FullName
10 * 10
(10 * 10).GetType().FullName
9 / 3
(9 / 3).GetType().FullName

When big integers trigger type promotion

Max value of a 64-bit signed integer (System.Int64)

[long]::MaxValue
[long]::MaxValue.GetType().FullName

Add a small number – looks harmless…

$max = 9223372036854775807
$max.GetType().FullName
$sum = $max + 2
$sum
$sum.GetType().FullName

Confirm lost of precision

$max,$sum | Foreach-Object{
    "{0:N0}" -f $_
}

Casting AFTER promotion can not fix lost precision

[ulong]$sum

Casting BEFORE promotion can not fix lost precision

[long]$sum = $max + 2

Compare with doing it “right” up front using an unsigned literal

$exact = 9223372036854775807ul + 2
$exact
$exact.GetType().FullName

Decimal: the precision bodyguard

Decimal keeps precision and throws on overflow

[decimal]::MaxValue
[decimal]::MaxValue.GetType().FullName
[decimal]::MaxValue + 1

Money and measurements: double vs decimal

Binary floating point: fast, but not always exact in base 10

0.1 + 0.2
(0.1 + 0.2) - 0.3

(0.1).ToString("G17")

Decimal literals for big math or money math

$priceTotal = 0.10d + 0.20d
$priceTotal
$priceTotal.GetType().FullName

Decimal behaves exactly here

$priceTotal - 0.30d

Wrap Up

• Large integer arithmetic can promote values to double,
• Doubles can cause precision loss when the result exceeds the integer range
• Casting after promotion does not restore the lost integer bits
• Use the correct literal suffixes (ul, l, d) to control the types
• Prefer decimal for money or other values that must be exact
• Always check your result types with .GetType().FullName

Related Posts