PostScript: Fixing rounding error in floating point number -
i trying make axis tick marks labeled numbers. working fine except last number generated axis 0.099999 instead of 0.01. have removed line drawing parts code below, left part generates numbers. these numbers below code: 1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.09999999 (should 0.01). there anyway round numbers or calculate numbers different way avoid rounding error?
/str 10 string def /count 0 def /totalunits 1 def /ticks 0.1 def totalunits ticks neg 0 { /loopnum exch def loopnum 10 str cvrs /stringnumber exch def % string representing number /count count 0.1 add def }
as commented, can use integers avoid quirk of floating-point numbers.
/str 10 string def /count 0 def /scaling .1 def 10 -1 1{ scaling mul 10 string cvs %dup = /stringnumber exch def /count count 1 add def }
this same rounding issue affects languages use binary floating-point representation numbers since fraction 1/10 has repeatinging pattern in binary representation. it's same effect gives .333333... 1/3 in decimal representation. summing these quantities loses bits bottom (input) values have been truncated.
Comments
Post a Comment