Understanding Geodesy

Learn the mathematics behind latitude/longitude calculations

🌐 The Spherical Earth Model

All calculations in GeoCalc Pro use a spherical Earth model, which treats our planet as a perfect sphere. While the Earth is actually slightly ellipsoidal (flattened at the poles), the spherical model is accurate enough for most purposes.

📊 Earth's Dimensions

  • Equatorial radius: ~6,378 km
  • Polar radius: ~6,357 km
  • Mean radius (used here): 6,371 km

Using a spherical model introduces errors typically up to 0.3%, which translates to about 3 meters per kilometer. For aviation, sailing, and most geographic applications, this is perfectly acceptable. For surveying or precision geodesy, ellipsoidal formulas (like Vincenty's) would be used instead.

⚠️

The 0.3% error is most pronounced when measuring distances that cross the equator or span large latitude differences.

🛤️ Great Circle vs Rhumb Line

There are two fundamentally different ways to travel between two points on Earth:

🔵 Great Circle (Orthodrome)

The shortest path between two points on a sphere. A great circle is any circle that divides the sphere into two equal hemispheres.

  • ✓ Shortest distance
  • ✓ Saves fuel/time
  • ✗ Bearing constantly changes
  • ✗ Harder to navigate manually

🟠 Rhumb Line (Loxodrome)

A path of constant bearing that crosses all meridians at the same angle. Appears as a straight line on a Mercator projection.

  • ✓ Constant compass heading
  • ✓ Easy to navigate
  • ✗ Longer distance
  • ✗ Spirals toward poles

📍 Real-World Example: London to New York

A great circle route from London to New York curves northward over the Atlantic, passing near Greenland. This saves about 4% compared to following a constant heading (rhumb line).

For aviation, that 4% translates to significant fuel savings. But for a sailing vessel in the 18th century, following a constant compass bearing was far more practical!

📐 The Haversine Formula

The haversine formula calculates great-circle distances between two points on a sphere. It's the most commonly used formula for this purpose due to its numerical stability.

The Formula

a = sin²(Δφ/2) + cos(φ₁) · cos(φ₂) · sin²(Δλ/2)

c = 2 · atan2(√a, √(1−a))

d = R · c

Where:

  • φ = latitude (in radians)
  • λ = longitude (in radians)
  • Δφ = difference in latitudes
  • Δλ = difference in longitudes
  • R = Earth's radius (6,371 km)
  • d = distance between points

📜 Historical Note

The name "haversine" comes from "half-versed-sine": hav(θ) = sin²(θ/2) = (1−cos(θ))/2

In the age of log tables, navigators used the haversine because it kept all values positive (no logarithms of negative numbers!) and avoided expensive addition operations. Roger Sinnott popularized this formula in 1984 in Sky & Telescope magazine, showing it could accurately calculate the tiny angular separation between the stars Mizar and Alcor.

JavaScript Implementation

const R = 6371e3; // Earth's radius in meters
const φ1 = lat1 * Math.PI/180;
const φ2 = lat2 * Math.PI/180;
const Δφ = (lat2-lat1) * Math.PI/180;
const Δλ = (lon2-lon1) * Math.PI/180;

const a = Math.sin(Δφ/2) * Math.sin(Δφ/2) +
          Math.cos(φ1) * Math.cos(φ2) *
          Math.sin(Δλ/2) * Math.sin(Δλ/2);
const c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a));

const distance = R * c; // in meters
✅

Why use haversine? It remains numerically stable even at very small distances (a few meters), where other formulas may suffer from rounding errors.

📏 Spherical Law of Cosines

A simpler alternative to the haversine formula, the spherical law of cosines can be expressed in a single line:

d = acos(sin(φ₁) · sin(φ₂) + cos(φ₁) · cos(φ₂) · cos(Δλ)) · R

Modern computers use IEEE 754 64-bit floating-point arithmetic, providing 15 significant figures of precision. This makes the law of cosines accurate down to distances of just a few meters.

Haversine vs Law of Cosines

Aspect Haversine Law of Cosines
Code complexity More complex Single line
Trig functions 7 trig + 2 sqrt 6 trig functions
Small distances Excellent Good (with 64-bit)
Speed Slightly faster Slightly slower
Historical use Navigation era Modern computing
💡

For most modern applications, both formulas give identical results. The haversine is preferred when you need guaranteed accuracy at very small distances or when working with lower-precision arithmetic.

⚡ Equirectangular Approximation

When performance matters more than precision, the equirectangular approximation offers a much faster alternative using simple Pythagorean geometry:

x = Δλ · cos(φₘ)

y = Δφ

d = R · √(x² + y²)

Where φₘ = (φ₁ + φ₂) / 2 is the mean latitude

This approximation treats the Earth as flat locally, projecting onto an equirectangular plane. It uses only one trig function and one square root, making it extremely fast.

⚠️

When NOT to use this:

  • Distances over ~100 km
  • Near the poles
  • When crossing the antimeridian

✅ Good Use Cases

  • Finding nearby restaurants within a city
  • Sorting thousands of points by distance
  • Real-time location filtering
  • Initial filtering before precise calculation

📊 Method Comparison

Choose the right method for your use case:

Recommended

🎯 Haversine

Best all-around choice for geographic applications.

Accuracy: Excellent Speed: Fast Range: Any distance

📏 Law of Cosines

Simple one-liner, good for modern systems.

Accuracy: Very Good Speed: Fast Range: > 1 meter

⚡ Equirectangular

Ultra-fast for short distances and filtering.

Accuracy: Good (local) Speed: Fastest Range: < 100 km

🔬 Vincenty

Ellipsoidal model for surveying precision.

Accuracy: Sub-mm Speed: Slower Range: Any distance

🧭 Understanding Bearings

A bearing is the direction from one point to another, measured as an angle from north (0°) clockwise to 360°.

N
0°
E
90°
S
180°
W
270°

Initial vs Final Bearing

On a great circle path, your compass heading constantly changes as you travel. The initial bearing is your heading when you start; the final bearing is your heading when you arrive.

Example: Flying from Baghdad (35°N, 45°E) to Osaka (35°N, 135°E) - both at the same latitude - you start heading 60° (ENE) but arrive heading 120° (ESE)!

Initial Bearing Formula

θ = atan2(sin(Δλ) · cos(φ₂), cos(φ₁) · sin(φ₂) − sin(φ₁) · cos(φ₂) · cos(Δλ))

Result is in radians; convert to degrees and normalize to 0-360°

🗺️ Rhumb Lines Explained

Also called loxodromes, rhumb lines were essential for navigation before GPS. A rhumb line maintains a constant compass bearing, making it easy to follow with just a compass.

🔑 Key Insight: Isometric Latitude

Rhumb line calculations use the inverse Gudermannian function to compute the "isometric latitude" (ψ):

ψ = ln(tan(π/4 + φ/2))

This represents the height on a Mercator projection map. Since Mercator projections stretch more at higher latitudes, rhumb lines appear as straight lines on these maps!

📍 Extreme Example: New York to Beijing

The rhumb line distance is about 30% longer than the great circle route - one of the most extreme differences possible for a practical route.

For aviation, this would mean burning significantly more fuel. But for historical sailing, the simplicity of following a constant bearing often outweighed the extra distance.

🌀

Polar Spiral: If you follow a rhumb line at a constant bearing (other than exactly N/S or E/W), you'll spiral toward one of the poles, never quite reaching it!

⚠️

Midpoint Confusion: A common mistake is to simply average the latitude and longitude of two points: ((lat1+lat2)/2, (lon1+lon2)/2). This actually gives the midpoint along a Rhumb Line, NOT the geodesic midpoint calculated by this tool.