Build a dataset for the platform¶
The web platform serves a near-optimal set: the near-optimal points with their costs and duals, detailed outputs for plotting, and a manifest describing them, including how the platform plots and labels them. This guide builds one from a PyPSA network. The format itself is described in Near-optimal set format.
You need the scripts extra, which adds MGA Surveyor and PyPSA:
uv sync --package mga-compass --extra scripts
The example dataset¶
mga-compass/scripts/generate_dataset.py builds the set of PyPSA’s model_energy example.
From the mga-compass directory:
uv run python scripts/generate_dataset.py --tol 5 --max-iter 120 --max-parallel 4
--tol, --max-iter and --max-parallel are required. The script uses Gurobi if a licence
is available and HiGHS otherwise, explores the 2%, 5% and 8% slack levels, and writes the set
to scripts/output/set/ and its archive to scripts/output/near-optimal-set.zip. The title
and description, names, units, plots and display units come from the dataset config
scripts/datasets/model_energy.json; see Dataset configs.
Then check the set and publish it:
uv run python scripts/publish_dataset.py --set-dir scripts/output/set --tag dataset-<date>
This checks the set, checks that its archive unpacks to the same set, and uploads the archive
as a new GitHub release (needs the GitHub CLI and write access to
the repository). It prints the DATASET_URL to give the server. Without --tag, it only
checks.
A dataset from your own network¶
Explore the network with MGA Surveyor; see Explore the near-optimal set of your PyPSA network.
Collect the results as a near-optimal set, with the unit of every property of interest and of the cost:
from mga_surveyor import collect_near_optimal_set nos, files = collect_near_optimal_set( "mga-output", units={"wind": "MW", "solar": "MW"}, cost_unit="EUR", cost_name="TOTEX", )
Every slack level needs its run summary, and every point its duals.
Extract the outputs you want to plot, as long tables: a
pointcolumn, one column per index set, and avaluecolumn.files[p]is the solved network of pointp. For example, installed capacity per technology:import pandas as pd import pypsa rows = [] for point, path in enumerate(files): n = pypsa.Network(path) capacity = n.generators.p_nom_opt rows += [(point, tech, value) for tech, value in capacity.items()] capacity = pd.DataFrame(rows, columns=["point", "technology", "value"])
generate_dataset.py’sextract_generation_and_capacityreads generation and capacity for all points at once, without loading each network fully.Add the outputs, their index sets and the plots. Write a dataset config, starting from
scripts/datasets/model_energy.json, and apply it as the scripts do:from pathlib import Path from dataset_config import DatasetConfig, write_set # in mga-compass/scripts config = DatasetConfig.load("scripts/datasets/my_network.json") nos, outputs = config.apply(nos, {"capacity": capacity}, {"technology": list(capacity["technology"].unique())}) write_set(nos, outputs, Path("my-output/set")) # also writes my-output/near-optimal-set.zip
Or set the manifest’s
title,description,sets,outputs,plotsanddisplayyourself and write the set withmga_format.write.Publish it with
publish_dataset.py, as for the example above.
Keep the values in the model’s units. To show them in other units, for example cost in
million euros and capacities in GW, give them a display entry in the dataset config: the
platform then shows the values times its factor.
To try the set, run the platform locally with DATA_DIR pointing at its
directory.