mga-format API

The mga-format package reads, writes and validates near-optimal sets. MGA Surveyor uses it to export a set, MGA Compass and its server to read one.

Near-optimal sets

The near-optimal set format, version 1: reading, validating and writing.

A near-optimal set is a directory of Parquet files and a manifest.json; the format is specified in docs/reference/near-optimal-set-format.md. This module holds the manifest’s models, reads a set while checking it against its manifest, and writes one.

In memory a set is a NearOptimalSet:

  • points: indexed by point; columns slack, kind, optionally key, one per PoI, and the cost column.

  • duals: indexed by point; one column per PoI.

  • sets: {name: DataFrame}, each with the members in a column named after the set, plus attribute columns.

Outputs are not loaded with the set: they can be large, so read_output() reads one output, optionally for a few points only.

mga_format.near_optimal_set.POINT = 'point'

The built-in index set whose members are the points.

mga_format.near_optimal_set.KEY = 'key'

Optional column of points.parquet: the producer’s identifier of the solve behind each point (for MGA Surveyor, the hash in its result file’s name).

exception mga_format.near_optimal_set.NearOptimalSetError[source]

Bases: ValueError

A near-optimal set that does not match the format or its own manifest.

class mga_format.near_optimal_set.Cost(*, name, unit, optimum)[source]

Bases: _Model

Parameters:
name: str
unit: str
optimum: float
model_config = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mga_format.near_optimal_set.Poi(*, name, unit, description='', definition)[source]

Bases: _Model

Parameters:
name: str
unit: str
description: str
definition: dict[str, dict[str, dict[str, float]]]

Surveyor’s dimensions entry: component class -> attribute -> {component: coefficient}.

model_config = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mga_format.near_optimal_set.SlackLevel(*, slack, norm, normalize, tol, tol_unit, vertex_method, objective_cuts, iterations, gap, converged)[source]

Bases: _Model

Parameters:
slack: float
norm: Literal['l1', 'l2', 'linf']
normalize: bool
tol: float
tol_unit: Literal['percent', 'model']
vertex_method: Literal['bb', 'cdd']
objective_cuts: bool
iterations: int
gap: float
converged: bool
model_config = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mga_format.near_optimal_set.Fingerprint(*, network)[source]

Bases: _Model

Parameters:

network (str)

network: str
model_config = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mga_format.near_optimal_set.IndexSet(*, description='')[source]

Bases: _Model

Parameters:

description (str)

description: str
model_config = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mga_format.near_optimal_set.Output(*, index, unit='', description='')[source]

Bases: _Model

Parameters:
index: list[str]
unit: str
description: str
model_config = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mga_format.near_optimal_set.Plot(*, type, output, x, stack=None)[source]

Bases: _Model

Parameters:
  • type (Literal['bar', 'timeseries', 'stacked_bar', 'stacked_timeseries'])

  • output (str)

  • x (str)

  • stack (str | None)

type: Literal['bar', 'timeseries', 'stacked_bar', 'stacked_timeseries']
output: str
x: str
stack: str | None
model_config = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mga_format.near_optimal_set.Display(*, unit, factor)[source]

Bases: _Model

How the web platform shows a quantity: its values times factor, in unit.

Parameters:
unit: str
factor: float
model_config = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mga_format.near_optimal_set.Manifest(*, format, version, title='', description='', created, software={}, cost, pois, slack_levels, fingerprint, files={}, sets={}, outputs={}, plots={}, display={})[source]

Bases: _Model

Parameters:
format: Literal['mga-near-optimal-set']
version: Literal[1]
title: str

What the set is of, in a few words, e.g. “PyPSA model_energy example”.

description: str

The model behind the set: what it covers and where it comes from.

created: datetime
software: dict[str, str]
cost: Cost
pois: list[Poi]
slack_levels: list[SlackLevel]
fingerprint: Fingerprint
files: dict[str, str]
sets: dict[str, IndexSet]
outputs: dict[str, Output]
plots: dict[str, Plot]
display: dict[str, Display]

Display units, keyed by a PoI, the cost column or an output.

property poi_names: list[str]
property slacks: list[float]
model_config = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mga_format.near_optimal_set.NearOptimalSet(manifest: 'Manifest', points: 'pd.DataFrame', duals: 'pd.DataFrame', sets: 'dict[str, pd.DataFrame]'=<factory>)[source]

Bases: object

Parameters:
manifest: Manifest
points: DataFrame
duals: DataFrame
sets: dict[str, DataFrame]
validate()[source]

Raise NearOptimalSetError unless the tables match the manifest.

Return type:

None

mga_format.near_optimal_set.sha256_file(path)[source]
Parameters:

path (Path)

Return type:

str

mga_format.near_optimal_set.read_manifest(directory)[source]

Read and validate manifest.json, refusing other formats and versions.

Parameters:

directory (str | Path)

Return type:

Manifest

mga_format.near_optimal_set.read(directory)[source]

Read a near-optimal set: verify its checksums, load its points, duals and index sets, and check them against the manifest. Outputs are read separately, with read_output().

Parameters:

directory (str | Path)

Return type:

NearOptimalSet

mga_format.near_optimal_set.read_output(directory, name, points=None)[source]

Read output name in its long form, for all points or only points.

Each point’s rows are one row group, so reading a few points reads only their row groups. They are found from the row groups’ statistics and read directly: a filtered dataset scan would find the same rows, but buffers far more memory on the way (some 200 MB more for a few points of a large output).

Parameters:
Return type:

DataFrame

mga_format.near_optimal_set.validate_output(directory, nos, name)[source]

Fully check output name: its columns, that every key is a member of its set, and that it is sorted by point with one row group per point. Reads the whole output, so read() does not do it.

Parameters:
Return type:

None

mga_format.near_optimal_set.write(directory, nos, outputs=None)[source]

Write nos and its outputs (long form, as in the format) to the new directory directory, record every file’s checksum in the manifest, and return that manifest. The set is validated first; directory must not exist.

Parameters:
Return type:

Manifest

Archives

A near-optimal set as one archive, for publishing and downloading.

A set is a directory with subdirectories (sets/, outputs/), but places such as GitHub releases hold only flat files. So a set is published as one zip archive of its directory: pack() creates it, unpack() restores the directory, and fetch() downloads and unpacks one. Both check the set with read() before handing it over.

mga_format.archive.pack(directory, archive)[source]

Write the set in directory to the zip file archive, after checking it. Files are stored uncompressed: Parquet files are already compressed.

Parameters:
Return type:

Path

mga_format.archive.unpack(archive, directory)[source]

Unpack the zip file archive into directory, replacing whatever is there, and return the set read from it. The set is unpacked and checked next to directory first, so a broken archive leaves directory as it was.

Parameters:
Return type:

NearOptimalSet

mga_format.archive.fetch(url, directory, timeout=600)[source]

Download the archive at url and unpack it into directory, replacing whatever is there, and return the set.

Parameters:
Return type:

NearOptimalSet