PostScript: Fixing rounding error in floating point number -


i trying make axis tick marks labeled numbers. working fine except last number generated axis 0.099999 instead of 0.01. have removed line drawing parts code below, left part generates numbers. these numbers below code: 1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.09999999 (should 0.01). there anyway round numbers or calculate numbers different way avoid rounding error?

/str 10 string def /count 0 def /totalunits 1 def /ticks 0.1 def  totalunits ticks neg 0 { /loopnum exch def loopnum 10 str cvrs /stringnumber exch def % string representing number /count count 0.1 add def } 

as commented, can use integers avoid quirk of floating-point numbers.

/str 10 string def  /count 0 def  /scaling .1 def   10 -1 1{     scaling mul      10 string cvs      %dup =     /stringnumber exch def      /count count 1 add def  }  

this same rounding issue affects languages use binary floating-point representation numbers since fraction 1/10 has repeatinging pattern in binary representation. it's same effect gives .333333... 1/3 in decimal representation. summing these quantities loses bits bottom (input) values have been truncated.


Comments

Popular posts from this blog

css - SVG using textPath a symbol not rendering in Firefox -

Java 8 + Maven Javadoc plugin: Error fetching URL -

order - Notification for user in user account opencart -