Skip to main content

Remaining Data Sources

This document tracks all data sources that have been identified but not yet imported into the VanSafe database.

Last Updated: October 16, 2025 Database Verified: October 16, 2025 (Direct Supabase query)

Database Verification Completed

This document reflects actual verified database status from direct Supabase queries, not assumptions from documentation. Total verified records: 3,080,374

Summary Status​

CategoryCompletedRemainingPriority
Mobile Coverage✅ COMPLETE (2.8M polygons)None Integration neededHigh ✅ Done
Bridge Restrictions✅ Complete (135,687)45 VIC rail overpassesMedium
Rest Areas✅ Partial (849 imported)Expand to other statesHigh
Route Segments❌ MISSINGNeed to import networkCRITICAL
Elevation Data❌ BLOCKEDCannot process without route_segmentsHigh
Weather Data❌ Not startedBOM/Open-Meteo pollingHigh
Live Traffic✅ NSW only (429 incidents)VIC, QLD, other statesMedium
Fuel Stations✅ Complete (7,429)Amenity enrichment (poor quality)Low
Protected Areas❌ Not startedCAPAD national parksMedium
Tourist POIs❌ Not startedOSM + government datasetsLow
Caravan Parks❌ Not startedWikiCamps/OSMLow

🔴 High Priority (Critical for MVP)​

0. Route Segments Network - CRITICAL BLOCKER​

Status: ❌ TABLE MISSING - Database verification shows route_segments table does not exist Impact: CRITICAL - Blocks elevation processing and terrain analysis Source: Geoscape National Roads or OpenStreetMap

What's Needed:

  • Import Australian road network as route segments
  • Expected: Millions of road segments
  • Required for elevation data attachment
  • Required for grade calculations

Data Sources:

Option 2: OpenStreetMap​

Implementation:

  • Check if import script exists: scripts/import-national-roads*.ts
  • May need to create route_segments table migration
  • Import road network data
  • Verify segment count (expect millions)

Estimated Time: 10-15 hours

  • 2-3 hours: Create/verify table schema
  • 4-6 hours: Download and process road network data
  • 4-6 hours: Import and verify segments

Blockers: ⚠️ CRITICAL - Elevation processing completely blocked without this. Must be completed before elevation work can proceed.

✅ 1. Mobile Coverage Data - COMPLETE!​

Status: ✅ COMPLETE - 2,814,508 polygons verified in database Impact: Safety - Users need to know coverage gaps Source: ACCC Mobile Infrastructure Report

Database Verification (October 16, 2025):

  • ✅ 2,814,508 coverage polygons imported
  • ✅ All major carriers: Telstra, Optus, Vodafone, TPG
  • ✅ Network types: 3G, 4G, 5G
  • ✅ Coverage types: outdoor, indoor, external antenna required
  • ✅ Migration 0011 already run successfully

What This Means: 🎉 No import needed - This was incorrectly marked as "not started" in documentation

Next Actions:

  • Import data Already complete
  • Integrate into mobile app UI
  • Display coverage on route maps
  • Show coverage warnings for routes
  • Add carrier selection filter

Time Saved: 3-4 hours (was previously estimated as high-priority work)

Related Documentation: Migration 0011

2. Rest Areas & Stopping Points - Expansion Needed​

Status: ✅ 849 rest areas already imported - Need to expand coverage Impact: High - Essential for trip planning Sources: Multiple government datasets

Database Verification (October 16, 2025):

  • ✅ 849 rest areas currently in database
  • ❓ Unknown which states/sources these represent
  • ✅ Import script exists and has been executed
  • ❌ Need to expand to additional states

What's Still Needed:

Determine Current Coverage​

# First: Check which states are already in database
# Query for data_source or state fields in rest_areas table

NSW Rest Areas (if not complete)​

QLD Rest Areas (450+ locations)​

  • Source: Queensland Transport and Main Roads
  • Format: CSV/GeoJSON
  • Status: Likely not imported

Other States​

  • VIC, WA, SA, TAS rest areas (various sources)
  • Status: Likely not imported

Database Table: rest_areas table exists

Estimated Time: 3-5 hours (reduced - script already exists)

  • 1 hour: Verify current coverage by state
  • 2-4 hours: Download and import missing states

Blockers: None - framework proven to work

3. Weather Data Integration​

Status: ❌ Not Started (framework exists) Impact: Safety - Weather warnings for route planning Source: Bureau of Meteorology (BOM) or Open-Meteo API

What's Needed:

  • URL: https://open-meteo.com/
  • Format: JSON REST API
  • Coverage: All of Australia
  • Features: 7-day forecasts, historical data
  • Cost: Free with attribution
  • Rate Limit: Generous for open source

Option 2: BOM FTP Archive​

  • URL: ftp://ftp.bom.gov.au/anon/gen/
  • Format: XML, JSON
  • Features: Official forecasts and warnings
  • Update: Hourly
  • Cost: Free

Database Tables:

  • ✅ weather_forecasts - Already exists (migration 0008)
  • ✅ weather_warnings - Already exists (migration 0008)

Implementation:

  • ❌ Create polling script: scripts/poll-weather-openmeteo.ts OR scripts/poll-bom-ftp.ts
  • ❌ Set up cron job (every 6 hours)
  • ❌ Add npm scripts

Estimated Time: 4-6 hours (Open-Meteo) or 8-10 hours (BOM FTP)

Blockers: None - tables exist, need polling implementation

4. VIC/QLD/Other States Live Traffic​

Status: ✅ NSW Complete (verified), ❌ Other states not started Impact: High - Real-time hazard awareness

Database Verification (October 16, 2025):

  • ✅ 429 total incidents in database
  • ✅ 429 active incidents
  • ✅ NSW complete implementation
  • ✅ Polling script working

What's Needed:

Victoria Live Traffic​

Queensland Traffic​

SA Road Closures​

National Freight Data Hub​

Implementation:

  • ✅ Framework exists (NSW implementation)
  • ❌ Extend polling script for additional states
  • ❌ Test and verify data quality

Estimated Time: 3-4 hours per state

Blockers: None - extend existing pattern

🟡 Medium Priority (Important for Complete Product)​

5. VIC Rail Overpasses (45 Missing Heights)​

Status: ❌ Not Started Impact: Safety - Critical for caravan routing Priority: HIGH for VIC, but only 45 structures

What's Needed:

  • Height clearances for 45 identified rail overpasses
  • Most are in metro Melbourne area

Data Sources:

Option 2: OpenStreetMap Cross-Reference​

  • Scripts exist: phase1-osm-manual-lookup.ts, phase2-osm-tourist-areas.ts
  • Status: Created but not executed
  • Method: Manual lookup for each structure

Option 3: Google Street View Triage​

  • Priority Structures: 5 highest-risk in metro Melbourne
  • Time: ~4 hours
  • Method: Visual inspection of height signs

Option 4: Contact Transport Victoria​

Estimated Time:

  • 2 hours: Manual VicRoads download
  • 1 hour: OSM cross-reference
  • 4 hours: Street View verification
  • OR 1-2 weeks: Wait for official data

See VIC Rail Overpasses for detailed action plan.

Blockers: Manual action required (can't automate due to API protection)

6. SA/NT Bridge Restrictions​

Status: ⚠️ NHVR coverage only Impact: Medium - These states have limited local data

South Australia:

  • Available: Adelaide City Council area only (limited scope)
  • Decision: Skip SA-specific import, rely on NHVR
  • Coverage: NHVR data likely provides better coverage

Northern Territory:

  • Available: No dedicated bridge dataset found
  • Alternative: Road Report NT (limited restriction info)
  • Coverage: NHVR data only
  • Action: May require contacting Department of Logistics and Infrastructure

Estimated Time: N/A - May not have additional data available

Blockers: Data availability - no statewide datasets identified

7. Elevation Data Processing - BLOCKED​

Status: ❌ BLOCKED - Cannot process without route_segments table Impact: High - Identifies steep grades for caravan safety

Database Verification (October 16, 2025):

  • ❌ route_segments table: Does not exist (returns error)
  • ✅ segment_elevations table: Exists but empty (0 rows)
  • ❓ SRTM tiles: May be downloaded locally, but cannot be used

Critical Issue: ⚠️ Previous documentation claiming "1.35% complete" and "4.3M segments" was incorrect. The route_segments table does not exist in the database.

What's Actually Needed:

Step 1: Import Route Segments (PREREQUISITE)

  • See section "0. Route Segments Network" above
  • Must complete BEFORE elevation processing can begin
  • Expected: Millions of road segments from Geoscape or OSM

Step 2: Then Process Elevation Data

  • Download SRTM 90m elevation tiles (if not already local)
  • Process elevation for each route segment
  • Calculate grades (% slope)
  • Identify steep sections

Estimated Time:

  • Route segments import: 10-15 hours (see above)
  • Elevation processing: 48-72 hours runtime (automated)
  • Total: Unknown until route_segments exists

Blockers: ⚠️ CRITICAL BLOCKER - route_segments table must be created and populated first. Elevation processing is completely blocked.

8. Protected Areas (National Parks)​

Status: ❌ Not Started Impact: Medium - Valuable for tourism Source: CAPAD (Australian Government)

What's Needed:

  • 14,575 national parks and protected areas
  • National Parks
  • Nature Reserves
  • Indigenous Protected Areas
  • Conservation Areas
  • Marine Protected Areas

Data Format: GeoJSON, WFS, Shapefile License: CC-BY (Free)

Source: https://www.environment.gov.au/fed/catalog

Implementation:

  • Direct GeoJSON download (no rate limits)
  • Similar to boundary import pattern
  • Add to amenities table or create protected_areas table

Estimated Time: 1-2 hours

Blockers: None - ready to implement

🟢 Low Priority (Nice to Have)​

9. Tourist Attractions & POIs​

Status: ❌ Not Started Impact: Low - Enhances user experience Sources: Multiple free and paid options

What's Needed:

OpenStreetMap Tourism Data

  • Coverage: National
  • Tags: tourism=attraction, viewpoint, museum, gallery, zoo, theme_park
  • Implementation: Extend existing OSM collection infrastructure
  • Estimated Time: 2-3 hours
  • Expected Records: 10,000+

NSW Points of Interest

Victorian Landmarks

  • Source: City of Melbourne Open Data
  • URL: https://data.melbourne.vic.gov.au/
  • Coverage: Melbourne metro
  • Format: Direct GeoJSON export
  • Estimated Time: 1 hour
  • Expected Records: 500+

ATDW API (Australian Tourism Data Warehouse)

  • URL: https://www.atdw.com.au/distributors
  • Coverage: National (50,000+ tourism products)
  • Cost: $165-$660 setup + possible ongoing fees
  • Features: Accommodation, attractions, tours, restaurants, events
  • Recommendation: Start with free trial
  • Estimated Time: 8-12 hours (SOAP/XML complexity)

Total Free Tourism POIs: ~15,000+ records Total Time: 6-8 hours for all free sources

Blockers: None - ready to implement

10. Caravan Parks & Campgrounds​

Status: ❌ Not Started Impact: Low - Nice for destination planning Sources: WikiCamps, OSM, state agencies

What's Needed:

WikiCamps Australia

  • Type: Commercial dataset
  • Option: Partnership or API access?
  • Coverage: Most comprehensive caravan-specific data

OpenStreetMap

  • Tags: tourism=camp_site, tourism=caravan_site
  • Coverage: Good but incomplete
  • Cost: Free

State Park Agencies

  • Sources: Various state governments
  • Coverage: Public campgrounds only

Estimated Time: 4-6 hours (OSM) or TBD (WikiCamps partnership)

Blockers: WikiCamps requires commercial arrangement

11. Fuel Station Amenity Enrichment​

Status: ✅ 7,429 stations complete, ⚠️ Amenity data very poor quality Impact: Low - Would improve existing data

Database Verification (October 16, 2025):

  • ✅ 7,429 fuel stations total
  • ⚠️ Worse quality than previously documented:
    • Stations with toilets: 15 (0.2%) - Down from claimed 1.9%
    • Stations with food: 21 (0.3%) - Down from claimed 2.6%
    • Stations with truck access: 1 (0.0%) - Critically low

What's Needed:

  • Scrape fuel station websites for amenity details
  • Crowdsource from VanSafe users
  • Import from additional data sources (e.g., WikiCamps)
  • Cross-reference with Google Places API (paid)

Estimated Time: Varies by method

  • Web scraping: 10-15 hours
  • User contributions: Ongoing
  • API integration: 3-4 hours

Blockers: May require paid API access (Google Places)

Implementation Priority Roadmap​

Week 1: Critical Blockers & Safety Features​

Total Time: ~17-26 hours

  1. Mobile Coverage ✅ COMPLETE (saves 3-4 hours)

    • Install dependencies
    • Download KML files
    • Import coverage data
    • Action: Integrate into mobile app UI

  2. Route Segments Import (10-15 hours) - NEW CRITICAL ITEM

    • Create/verify route_segments table
    • Download Geoscape or OSM road network
    • Import and verify segments
    • PREREQUISITE for elevation processing

  3. Rest Areas Expansion (3-5 hours)

    • Check current state coverage
    • Download missing state datasets
    • Import additional rest areas

  4. Weather Integration (4-6 hours)

    • Choose API (Open-Meteo recommended)
    • Create polling script
    • Set up cron automation

  5. VIC Rail Overpasses (2-3 hours)

    • Manual VicRoads download
    • Import 45 missing heights
    • OR Contact Transport Victoria

Week 2: Live Data Expansion & Elevation Processing​

Total Time: ~8-10 hours (+ automated processing time)

  1. Victoria Live Traffic (3-4 hours)

    • Extend existing polling script
    • Test and verify

  2. Queensland Traffic (3-4 hours)

    • Add QLD endpoints
    • Test and verify

  3. Protected Areas (1-2 hours)

    • Download CAPAD data
    • Import national parks

  4. Start Elevation Processing (48-72 hours runtime)

    • After route_segments imported
    • Process SRTM elevation data
    • Monitor progress and quality

Week 3+: Enhanced Features (Low Priority)​

Total Time: ~10-15 hours

  1. Tourism POIs (6-8 hours)

    • OSM tourism data
    • NSW POI dataset
    • Melbourne landmarks

  2. Caravan Parks (4-6 hours)

    • OSM campgrounds
    • State datasets

  3. Fuel Amenity Enrichment (Ongoing)

    • User contributions
    • Optional: API integration

Estimated Total Remaining Work​

Updated after database verification - October 16, 2025

By Priority​

  • Critical (Blockers): 10-15 hours (Route segments import)
  • High Priority: 7-11 hours (Mobile coverage done, rest areas, weather)
  • Medium Priority: 8-10 hours (Live traffic expansion, VIC bridges, protected areas)
  • Low Priority: 10-15 hours (Tourism, caravan parks, amenities)
  • Total: 35-51 hours (down from 35-50 hrs, but route_segments adds 10-15 hrs)

By Category​

  • Critical Infrastructure: 10-15 hours (Route segments - required for elevation)
  • Safety Critical: 7-11 hours (Mobile coverage complete, weather, rest areas expansion)
  • Data Completeness: 8-10 hours (Live traffic, VIC bridges, protected areas)
  • User Experience: 10-15 hours (Tourism, caravan parks, amenities)

Major Changes from Previous Estimate​

  • ✅ Mobile coverage: 3-4 hours → COMPLETE (saves 3-4 hours)
  • ✅ Rest areas: 6-8 hours → 3-5 hours (script exists, 849 imported)
  • ❌ Route segments: NEW - 10-15 hours (critical blocker discovered)
  • ❌ Elevation: In progress → BLOCKED (cannot proceed without route_segments)

Cost Analysis​

  • Free Data Sources: $0 (all high/medium priority)
  • Optional Paid: $165-$660 (ATDW tourism data)
  • API Services: $0-$50/month (weather, if using paid tier)

Recommendation: Complete all high-priority items first (Week 1) for MVP launch.

Dependencies & Blockers​

No Blockers (Ready to Implement)​

  • ✅ Mobile coverage - COMPLETE
  • ✅ Rest areas expansion (framework exists)
  • ✅ Weather integration (tables exist)
  • ✅ Live traffic (VIC/QLD) (NSW framework complete)
  • ✅ Protected areas (data available)
  • ✅ Tourism POIs (OSM) (can collect immediately)
  • ⚠️ Route segments (data available, but requires significant import work)

Manual Action Required​

  • ⚠️ VIC rail overpasses (manual download or wait for official response)
  • ⚠️ SA/NT bridges (may not have data available)

Commercial/Partnership Required​

  • 💰 WikiCamps (caravan parks)
  • 💰 ATDW (premium tourism data)
  • 💰 Google Places (amenity enrichment)

Critical Blockers​

  • ❌ Route segments table missing - Must be imported before elevation processing can begin
  • ❌ Elevation processing BLOCKED - Cannot proceed without route_segments table

Next Actions​

Updated after database verification - October 16, 2025

  1. Immediate (This Week):

    • Install mobile coverage dependencies ✅ Already complete
    • Integrate mobile coverage into mobile app (data already in database)
    • Import route_segments table (critical blocker for elevation)
    • Check current rest areas coverage by state
    • Download additional rest areas datasets if needed
    • Set up weather polling (Open-Meteo)
    • Download VicRoads bridge data OR send email to Transport Victoria

  2. Short-term (Next 2 Weeks):

    • Import mobile coverage ✅ Already complete (2.8M polygons)
    • Start elevation processing (after route_segments imported)
    • Extend live traffic to VIC/QLD
    • Import protected areas
    • Monitor elevation processing progress

  3. Long-term (1+ Month):

    • Tourism POIs
    • Caravan parks
    • Fuel station amenity enrichment
    • Evaluate paid data sources