Data Maintenance & Weekly Refresh

wikipediaGATN is designed to provide an up-to-date view of the global air transportation network. To achieve this, the repository includes automated processes for refreshing the data and re-generating network analysis files.

Automated Weekly Refresh

A GitHub Action, defined in .github/workflows/refresh_data.yml, runs automatically every Sunday at 00:00 UTC.

This workflow performs the following steps:

  1. Incremental Update: It runs python -m wikipediaGATN.refresh_airport_data --target all, which checks the “last modified” timestamp of every airport page on Wikipedia against the local version. Only pages that have changed since the last crawl are re-fetched.

  2. Network Re-generation: It runs python -m wikipediaGATN.result_processing_network to rebuild the global connection lists, sparse adjacency matrices, and graph-theoretic export files (GraphML, GEXF, DOT).

  3. Visualisation Update: It runs python -m wikipediaGATN.visualise_gatn to update the interactive Plotly HTML maps and globes.

  4. Commit & Push: If any data has changed, the bot automatically commits and pushes the updated files in data/public/ back to the main branch.

Manual Data Refresh

You can trigger a data refresh manually in two ways:

Via GitHub Actions UI

If you have write access to the repository:

  1. Navigate to the Actions tab on GitHub.

  2. Select the Weekly Data Refresh workflow.

  3. Click the Run workflow dropdown and select Run workflow.

Via Command Line

If you have cloned the repository and installed the package in development mode, you can run the same pipeline locally:

# 1. Update airport metadata JSONs
python -m wikipediaGATN.refresh_airport_data --target all

# 2. Rebuild network files (CSVs, matrices, graphs)
python -m wikipediaGATN.result_processing_network

# 3. Update HTML visualisations
python -m wikipediaGATN.visualise_gatn

Adding New Airports

The current global dataset was initialized from a full crawl starting from Winnipeg (YWG). If you discover a missing airport, you can add it to the network by running a local search with that airport as a seed:

from wikipediaGATN.wikipedia_network_level import iterate_search_until_distance_N
from wikipediaGATN.result_processing_airports import export_all_airport_data

# 1. Scrape the new airport and its immediate neighbors
iterate_search_until_distance_N("NEW_IATA_CODE", dist=1, verbose=True)

# 2. Integrate the new JSONs into the public repository
export_all_airport_data(use_new_data=True, verbose=True)

After integrating new data, follow the steps in the Manual Data Refresh section to rebuild the aggregate network files.