Near-optimal set format¶
Version 1. The format in which MGA Surveyor hands a computed near-optimal set to MGA Compass (the web platform and the Python API).
A near-optimal set is a directory of Parquet files and one manifest.json:
my-set/
├── manifest.json # what the files are: PoIs, cost, run settings and results, checksums
├── points.parquet # one row per point: slack, kind, key, PoI values, cost
├── duals.parquet # one row per point: duals of the PoIs
├── sets/ # optional: index sets of the outputs, one file each
│ ├── snapshot.parquet
│ └── technology.parquet
└── outputs/ # optional: detailed outputs, one variable per file
├── capacity.parquet
└── generation.parquet
Surveyor writes manifest.json (except its presentation part), points.parquet and
duals.parquet. The index sets and outputs and
the presentation part of the manifest are added for the web platform.
points.parquet and duals.parquet are wide: one row per point, one column per PoI.
Output files are long: one row per value, one column per index set.
All values are in the model’s units, as recorded in the manifest. Nothing is rescaled.
points.parquet¶
One row per near-optimal point, in no particular order.
Column |
Type |
Meaning |
|---|---|---|
|
int64 |
Identifier, unique, |
|
float64 |
The slack \(\epsilon\) of the level the point was found at; |
|
string |
How it was found: |
|
string, nullable |
Optional. The producer’s identifier for the solve behind the point, so its full result can be found again. Keys need not be unique. |
one column per PoI |
float64 |
The point’s PoI values \(z\), named and ordered as in the manifest’s |
the cost column |
float64 |
The point’s cost \(\nu(z)\): the cheapest system cost with the PoIs fixed at \(z\). Named by the manifest’s |
MGA Surveyor writes key as the hash in a result file’s name: a box point at slack
\(\epsilon\) comes from slack_<ε>/mga_furthest_result_<key>.nc and a nearest point from
slack_<ε>/mga_closest_result_<key>.nc, under the run’s output directory. The optimum has no
key; its result is optimal.nc. These names are Surveyor’s convention, not part of the format.
A point found at slack \(\epsilon'\) is near-optimal for every slack \(\epsilon \ge \epsilon'\).
To work with the near-optimal set at slack \(\epsilon\), use all points with
slack \(\le \epsilon\).
duals.parquet¶
Required. One row per point, with the same point identifiers as points.parquet. Every
point must have all its duals: a set whose duals are incomplete cannot be written, and is
refused when read.
Column |
Type |
Meaning |
|---|---|---|
|
int64 |
As in |
one column per PoI |
float64 |
The dual \(\mu_i\) of fixing that PoI: a subgradient of \(\nu\) at the point, in cost units per PoI unit. Every value must be a real number: no NaN. |
The duals give the cost lower bound \(\nu(z) \ge \nu(z^{(k)}) + \mu^{(k)\top}(z - z^{(k)})\).
manifest.json¶
The manifest has three parts:
Core (required): written by Surveyor. Describes the points and their duals.
Outputs (optional):
setsandoutputs, present when the set has detailed outputs.Presentation (optional):
plotsanddisplay, used only by the web platform.
A complete example with all three parts:
{
"format": "mga-near-optimal-set",
"version": 1,
"title": "PyPSA model_energy example",
"description": "One electricity node and one hydrogen node over 2019, at 9-hourly resolution.",
"created": "2026-11-06T12:00:00+00:00",
"software": {"mga-surveyor": "0.1.0", "pypsa": "1.3.0"},
"cost": {"name": "TOTEX", "unit": "EUR", "optimum": 3899079890.0},
"pois": [
{
"name": "wind",
"unit": "MW",
"description": "Installed wind capacity.",
"definition": {"Generator": {"p_nom": {"wind": 1}}}
}
],
"slack_levels": [
{
"slack": 0.05,
"norm": "linf",
"normalize": true,
"tol": 5.0,
"tol_unit": "percent",
"vertex_method": "bb",
"objective_cuts": true,
"iterations": 31,
"gap": 4.45,
"converged": true
}
],
"fingerprint": {"network": "<sha256>"},
"files": {
"points.parquet": "<sha256>",
"duals.parquet": "<sha256>",
"sets/snapshot.parquet": "<sha256>",
"sets/technology.parquet": "<sha256>",
"outputs/generation.parquet": "<sha256>",
"outputs/capacity.parquet": "<sha256>"
},
"sets": {
"snapshot": {"description": "Time steps of the model."},
"technology": {"description": "Generation and storage technologies."}
},
"outputs": {
"generation": {"index": ["point", "snapshot", "technology"], "unit": "MW",
"description": "Electricity generation per technology."},
"capacity": {"index": ["point", "technology"], "unit": "MW",
"description": "Installed capacity per technology."}
},
"plots": {
"generation": {"type": "stacked_timeseries", "output": "generation",
"x": "snapshot", "stack": "technology"},
"capacity": {"type": "bar", "output": "capacity", "x": "technology"}
},
"display": {
"wind": {"unit": "GW", "factor": 0.001},
"TOTEX": {"unit": "bn EUR", "factor": 1e-9},
"generation": {"unit": "GW", "factor": 0.001}
}
}
Field |
Part |
Meaning |
|---|---|---|
|
core |
Always |
|
optional |
What the set is of, in a few words. Empty if not given. |
|
optional |
The model behind the set: what it covers and where it comes from. Empty if not given. |
|
core |
When the set was exported, ISO 8601 with time zone. |
|
core |
Versions of the packages that produced it. |
|
core |
Name of the cost column in |
|
core |
Unit of the cost. |
|
core |
The optimal cost \(\nu^*\). |
|
core |
The properties of interest, in column order: name, unit, description, and the definition as Surveyor’s |
|
core |
One entry per slack level that was explored: the settings of the run ( |
|
core |
Identifies the input network, so the set cannot be used with a different one; see Network fingerprint. The PoIs need no fingerprint: their full definitions are in |
|
core |
sha256 of every other file in the directory, keyed by its path relative to the directory, including those under |
|
outputs |
The index sets; see In the manifest. |
|
outputs |
The outputs; see In the manifest. |
|
presentation |
The plots the web platform shows; see Presentation part (web platform only). |
|
presentation |
The units the web platform shows quantities in; see Presentation part (web platform only). |
A reader ignores the optional parts it does not use: the Python API needs only the core, the web platform needs all three.
Index sets and outputs¶
Detailed outputs, such as the generation of every technology at every time step, are variables over index sets. The sets are defined once and shared by all outputs.
Index sets¶
Each index set is one file, sets/<name>.parquet:
Column |
Type |
Meaning |
|---|---|---|
|
string, integer or timestamp |
The set’s members: unique, in display order. |
further columns |
any |
Optional attributes of the members, e.g. |
For example, sets/technology.parquet:
technology |
label |
color |
|---|---|---|
onwind |
Onshore Wind |
#235ebc |
solar |
Solar |
#f9d002 |
point is a built-in set whose members are the point column of points.parquet; it has no
file of its own.
Outputs¶
Each output is one variable, stored in outputs/<name>.parquet in long form:
Column |
Type |
Meaning |
|---|---|---|
one column per index set |
the set’s type |
The keys, named after the sets, in the order the manifest gives. |
|
float64 or float32 |
The value. float32 keeps about 7 significant digits in half the space: enough for large outputs such as time series. |
For example, outputs/generation.parquet:
point |
snapshot |
technology |
value |
|---|---|---|---|
0 |
2013-01-01 00:00 |
onwind |
1234.5 |
0 |
2013-01-01 00:00 |
solar |
0.0 |
Rules:
Every output’s index starts with
point, and every other set it uses is declared in the manifest’ssets.Every key is a member of its set. A missing combination means the value does not exist (for example, a technology that is not built at a node), not zero.
Rows are sorted by
point, and each point’s rows form one Parquet row group, so that a reader can load the outputs of a few points without reading the whole file.
In the manifest¶
See the complete example above.
Field |
Meaning |
|---|---|
|
{name: { |
|
{name: { |
Presentation part (web platform only)¶
Field |
Meaning |
|---|---|
|
{name: { |
|
{name: { |
Network fingerprint¶
fingerprint.network is the sha256 of the network’s input data:
the snapshots and their weightings;
every component’s static attributes marked as input in PyPSA;
every component’s time series marked as input in PyPSA.
Optimization results, such as optimized capacities, dispatch and prices, are left out, so solving the network again does not change the fingerprint, while changing any cost, limit or time series does. Two sets with the same fingerprint were computed from the same model.
Archive¶
To publish a set as a single file, for example as a GitHub release asset, it is packed into a
zip archive, conventionally named near-optimal-set.zip, that holds the directory’s files at
their relative paths (manifest.json, points.parquet, sets/…, outputs/…). Unpacking it
gives the directory back, and the set is checked against the manifest’s checksums as when
reading a directory.
Sets are immutable: a new set is a new archive under a new name or release, never an overwrite of an old one.
Functions¶
The mga-format package implements the format; see mga-format API.
Function |
Does |
|---|---|
Write a set, with its outputs, to a new directory. Refuses a set that breaks a rule above, such as missing duals. |
|
Read a set’s manifest, points, duals and index sets, after checking every file’s checksum. Outputs are not loaded. |
|
Read one output, optionally only for some points; reads only those points’ row groups. |
|
Pack a set directory into an archive, and unpack one. |
|
Download an archive from a URL and unpack it. |
MGA Surveyor writes a set with export_near_optimal_set(), or
collects it in memory with collect_near_optimal_set() to add
outputs first.