NoteTab Numeric Accuracy


I’ve been working on something to help me with some logging activity in NoteTab and I’ll be posting a variety of useful bits and pieces from this in both this micro blog (FlagIT) and my main blog (RebootIT).  As part of this I was looking at producing some date related clips which necessitated looking at some mathematical clips and notably one to determine if a number is an Integer.  NoteTab is a text editor and numeric manipulations are definitely not its forte, so I needed to produce a clip function to do this.

Integers are whole numbers.  They have no decimal part.  Fortunately the ^$Calc()$ function does expose a few mathematical functions to a clipper such as myself and one of these is INT.

As an aside at this point the range of mathematical functions available is quite deeply hidden.  The clip help file does provide one link back into the main NoteTab help file where there is a section called “Calculate in NoteTab” which I’d recommend taking a look at.  To use the functions in there you’ll need to understand the function and the number of operands.  INT has one operand (or parameter) that is passed to it,  and this is illustrated below.

So if we take a number and evaluate it to X decimal places, then subtract its integer part (i.e. anything to the left of the decimal point), we are left with the decimal part.  This remainder will be zero if the original number was an integer and non-zero if the original number was a non-integer.

e.g. (all to 3 decimal places)

  • 3.000 – 3.000 = 0.000 => Integer
  • 3.123 – 3.000 = 0.123 => Non-Integer
  • (-3.123)-(-3.000) = -0.123 => Non-Integer

To ensure that no matter what number was being passed, I had to be sure to evaluate to an appropriate number of decimal places otherwise I could get into trouble…

e.g. (all to 3 decimal places)

  • 3.0001 would give 3.000 – 3.000 = 0.000 => Integer … which is wrong.

Therefore if the level of accuracy is equal to the length of the number of characters being passed in I’m always assured of the correct level of accuracy.


3 would use an accuracy of 1 decimal place

  • 3.0 – 3.0 = 0.0 => Integer

3.123 would use an accuracy of 5 decimal places

  • 3.12300 – 3.00000 = 0.12300 => Non-Integer

So the clip function for IsInteger becomes the following:

^!Set %DPofAccuracy%=^$StrSize("^&")$
^!If ^$Calc(INT(^&);^%DPofAccuracy%)$=^$Calc(^&;^%DPofAccuracy%)$ END

This however got me to thinking.  Every programming language using numeric variables specifies a level of accuracy in decimal places, significant figures, etc. based upon the memory allocation internally managed for the variable.  I couldn’t find anything stating what this was in NoteTab (which isn’t to say it doesn’t exist) and so I thought I’d write a quick clip to work it out.

H="Numeric Variable Accuracy"
^!Set %Accuracy%=0
^!If ^%Accuracy%=0 SKIP
^!Set %CurrentValue%=1.^$StrFill("0";^$Calc(^%Accuracy%-1)$)$1
^!Set %Output%=^%Output^%NL%(^%Accuracy%) ^%CurrentValue% : ^$Calc(^%CurrentValue%;^%Accuracy%)$
^!Inc %Accuracy%
^!If ^%Accuracy%=20 FINISH
^!Goto LOOP
^!Info ^%Output%

The clip simply builds an ever smaller decimal part and then displays it to a specified number of decimal places.  I set it to display the results after 20 iterations from a starting point of no decimal places and these are the results (structured here in a nice table format for easier reading off the page).

Decimal Places Value Evaluation
0 1 1
1 1.1 1.1
2 1.01 1.01
3 1.001 1.001
4 1.0001 1.0001
5 1.00001 1.00001
6 1.000001 1.000001
7 1.0000001 1.0000001
8 1.00000001 1.00000001
9 1.000000001 1.000000001
10 1.0000000001 1.0000000001
11 1.00000000001 1.00000000001
12 1.000000000001 1.000000000001
13 1.0000000000001 1.0000000000001
14 1.00000000000001 1.00000000000001
15 1.000000000000001 1.000000000000001
16 1.0000000000000001 1.0000000000000000
17 1.00000000000000001 1.00000000000000000
18 1.000000000000000001 1.000000000000000000
19 1.0000000000000000001 1.0000000000000000000

As you can see at 16 decimal places there is a difference between the original value and the evaluated value. Therefore the IsInteger function (as well as any other mathematical calculations using NoteTab directly) will be subject to a 15 decimal places of accuracy limit.

Now this means that we need to ensure that the maximum number of decimal places used is also limited for the IsInteger function as if we then try to use a number with a total character length that exceeds 15 we start to get unpredictable results.

For example during the following evaluation^$IsInteger(10203040506070.1)$ the function actually ends up evaluating the following equality in the ^!If statement (line 4 of the clip).


Whilst it happens to give the right result there is scope here for failure.

Therefore we need to amend the clip to only allow up to 15 decimal places of accuracy.

^!Set %DPofAccuracy%=^$Calc(MIN(^$StrSize("^&")$;15))$
^!If ^$Calc(INT(^&);^%DPofAccuracy%)$=^$Calc(^&;^%DPofAccuracy%)$ END

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: