Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Floating point errors can cause incorrect digits after position 10 #121

Closed
drinckes opened this issue Nov 15, 2017 · 0 comments
Closed

Floating point errors can cause incorrect digits after position 10 #121

drinckes opened this issue Nov 15, 2017 · 0 comments
Assignees
Labels

Comments

@drinckes
Copy link
Contributor

Floating point errors can cause incorrect code to generate. This depends on the language, but as an example we noticed that encoding latitude 1, longitude 1 produces an X for the 11th digit.

The 10 digit code latitude and longitude ranges are the same - 1.0 to 1.000125. In the 11 digit sub-grid, the characters are assigned as:

Longitude 1.0 to 1.0003125 Longitude 1.0003125 to 1.000625 Longitude 1.000625 to 1.0009375 Longitude 1.0009375 to 1.000125
Latitude 1.0001 to 1.000125 R V W X
Latitude 1.000075 to 1.0001 J M P Q
Latitude 1.000050 to 1.000075 C F G H
Latitude 1.000025 to 1.000050 6 7 8 9
Latitude 1.0 to 1.000025 2 3 4 5

So for latitude 1.0, longitude 1.0 we want to get 2. But we are getting X.

This seems to be a problem in the encodeGrid functions where we try to work out which column the longitude is in by getting the remainder of dividing the longitude by the grid size (0.000125).

You can check this in Javascript by doing:
1.0 % 0.000125

It should be zero, but in my browser I'm seeing 0.00012499999999997919 which is nonsense.

Instead of directly comparing values in that function, we may need to use "close enough" comparisons, where if numbers are within 1E-10 of each other that's good enough.

Another strategy is instead of doing the remainder of 181.5 % 0.000125, we ditch the degrees and use 0.5 % 0.000125.

The solution in each case is probably language dependent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant