mga-format API¶
The mga-format package reads, writes and validates near-optimal sets.
MGA Surveyor uses it to export a set, MGA Compass and its server to read one.
Near-optimal sets¶
The near-optimal set format, version 1: reading, validating and writing.
A near-optimal set is a directory of Parquet files and a manifest.json; the
format is specified in docs/reference/near-optimal-set-format.md. This module
holds the manifest’s models, reads a set while checking it against its manifest,
and writes one.
In memory a set is a NearOptimalSet:
points: indexed bypoint; columnsslack,kind, optionallykey, one per PoI, and the cost column.duals: indexed bypoint; one column per PoI.sets: {name: DataFrame}, each with the members in a column named after the set, plus attribute columns.
Outputs are not loaded with the set: they can be large, so read_output()
reads one output, optionally for a few points only.
- mga_format.near_optimal_set.POINT = 'point'¶
The built-in index set whose members are the points.
- mga_format.near_optimal_set.KEY = 'key'¶
Optional column of points.parquet: the producer’s identifier of the solve behind each point (for MGA Surveyor, the hash in its result file’s name).
- exception mga_format.near_optimal_set.NearOptimalSetError[source]¶
Bases:
ValueErrorA near-optimal set that does not match the format or its own manifest.
- class mga_format.near_optimal_set.Cost(*, name, unit, optimum)[source]¶
Bases:
_Model- model_config = {'extra': 'forbid'}¶
Configuration for the model, should be a dictionary conforming to [
ConfigDict][pydantic.config.ConfigDict].
- class mga_format.near_optimal_set.Poi(*, name, unit, description='', definition)[source]¶
Bases:
_Model- Parameters:
- definition: dict[str, dict[str, dict[str, float]]]¶
Surveyor’s
dimensionsentry: component class -> attribute -> {component: coefficient}.
- model_config = {'extra': 'forbid'}¶
Configuration for the model, should be a dictionary conforming to [
ConfigDict][pydantic.config.ConfigDict].
- class mga_format.near_optimal_set.SlackLevel(*, slack, norm, normalize, tol, tol_unit, vertex_method, objective_cuts, iterations, gap, converged)[source]¶
Bases:
_Model- Parameters:
- norm: Literal['l1', 'l2', 'linf']¶
- tol_unit: Literal['percent', 'model']¶
- vertex_method: Literal['bb', 'cdd']¶
- model_config = {'extra': 'forbid'}¶
Configuration for the model, should be a dictionary conforming to [
ConfigDict][pydantic.config.ConfigDict].
- class mga_format.near_optimal_set.Fingerprint(*, network)[source]¶
Bases:
_Model- Parameters:
network (str)
- model_config = {'extra': 'forbid'}¶
Configuration for the model, should be a dictionary conforming to [
ConfigDict][pydantic.config.ConfigDict].
- class mga_format.near_optimal_set.IndexSet(*, description='')[source]¶
Bases:
_Model- Parameters:
description (str)
- model_config = {'extra': 'forbid'}¶
Configuration for the model, should be a dictionary conforming to [
ConfigDict][pydantic.config.ConfigDict].
- class mga_format.near_optimal_set.Output(*, index, unit='', description='')[source]¶
Bases:
_Model- model_config = {'extra': 'forbid'}¶
Configuration for the model, should be a dictionary conforming to [
ConfigDict][pydantic.config.ConfigDict].
- class mga_format.near_optimal_set.Plot(*, type, output, x, stack=None)[source]¶
Bases:
_Model- Parameters:
- type: Literal['bar', 'timeseries', 'stacked_bar', 'stacked_timeseries']¶
- model_config = {'extra': 'forbid'}¶
Configuration for the model, should be a dictionary conforming to [
ConfigDict][pydantic.config.ConfigDict].
- class mga_format.near_optimal_set.Display(*, unit, factor)[source]¶
Bases:
_ModelHow the web platform shows a quantity: its values times
factor, inunit.- model_config = {'extra': 'forbid'}¶
Configuration for the model, should be a dictionary conforming to [
ConfigDict][pydantic.config.ConfigDict].
- class mga_format.near_optimal_set.Manifest(*, format, version, title='', description='', created, software={}, cost, pois, slack_levels, fingerprint, files={}, sets={}, outputs={}, plots={}, display={})[source]¶
Bases:
_Model- Parameters:
format (Literal['mga-near-optimal-set'])
version (Literal[1])
title (str)
description (str)
created (datetime)
cost (Cost)
slack_levels (list[SlackLevel])
fingerprint (Fingerprint)
- format: Literal['mga-near-optimal-set']¶
- version: Literal[1]¶
- created: datetime¶
- slack_levels: list[SlackLevel]¶
- fingerprint: Fingerprint¶
- model_config = {'extra': 'forbid'}¶
Configuration for the model, should be a dictionary conforming to [
ConfigDict][pydantic.config.ConfigDict].
- class mga_format.near_optimal_set.NearOptimalSet(manifest: 'Manifest', points: 'pd.DataFrame', duals: 'pd.DataFrame', sets: 'dict[str, pd.DataFrame]'=<factory>)[source]¶
Bases:
object- validate()[source]¶
Raise
NearOptimalSetErrorunless the tables match the manifest.- Return type:
None
- mga_format.near_optimal_set.read_manifest(directory)[source]¶
Read and validate
manifest.json, refusing other formats and versions.
- mga_format.near_optimal_set.read(directory)[source]¶
Read a near-optimal set: verify its checksums, load its points, duals and index sets, and check them against the manifest. Outputs are read separately, with
read_output().- Parameters:
- Return type:
- mga_format.near_optimal_set.read_output(directory, name, points=None)[source]¶
Read output
namein its long form, for all points or onlypoints.Each point’s rows are one row group, so reading a few points reads only their row groups. They are found from the row groups’ statistics and read directly: a filtered dataset scan would find the same rows, but buffers far more memory on the way (some 200 MB more for a few points of a large output).
- mga_format.near_optimal_set.validate_output(directory, nos, name)[source]¶
Fully check output
name: its columns, that every key is a member of its set, and that it is sorted by point with one row group per point. Reads the whole output, soread()does not do it.- Parameters:
nos (NearOptimalSet)
name (str)
- Return type:
None
Archives¶
A near-optimal set as one archive, for publishing and downloading.
A set is a directory with subdirectories (sets/, outputs/), but places
such as GitHub releases hold only flat files. So a set is published as one zip
archive of its directory: pack() creates it, unpack() restores the
directory, and fetch() downloads and unpacks one. Both check the set with
read() before handing it over.
- mga_format.archive.pack(directory, archive)[source]¶
Write the set in
directoryto the zip filearchive, after checking it. Files are stored uncompressed: Parquet files are already compressed.