Make Test Dataset

When you encounter unexpected errors in the power-grid-model, you would like certainly to report the issue and debug (maybe by another developer) the calculation core with certain dataset. To make this possible, we have implemented a generic mechanism to export/import the dataset to/from JSON files, and to debug the calculation core in both Python and C++ with the test dataset.

In this notebook we will learn how test datasets are made in this repository, including:

  • Structure of validation test datasets in this repository

  • Format of test datasets (JSON)

  • Use of helper functions to save and load the datasets

Structure of Validation Datasets

All validation test datasets are located in the tests/data folder. The structure of the folder is as follows:

data
   |
   |
   - power_flow
             |
             - power_flow_testset_1
             - power_flow_testset_2
             - ...
   - state_estimation
             |
             - state_estimation_testset_1
             - state_estimation_testset_2
             - ...

The testsets are separated in two types of calculations: power_flow and state_estimation. In each folder there are subfolders for individual testset. The test datasets are used in both Python and C++ unit tests. Therefore, once you create extra test datasets in the folder, you can debug the program in both Python and C++.

Individual Dataset

An individual dataset folder (in either power_flow or state_estimation) will consists of following files:

  • params.json: calculation parameters, mandatory

  • input.json: input data, mandatory

  • sym_output.json: reference symmetric output

  • asym_output.json: reference asymmetric output

  • update_batch.json: update batch data, mandatory if sym_output_batch.json or asym_output_batch.json exists.

  • sym_output_batch.json: reference symmetric batch output

  • asym_output_batch.json: reference asymmetric batch output

The params.json and input.json are always needed. The test program (in Python and C++) will detect other files and instantiate relevant test calculations. For example, if sym_output.json exists, the test program will run a one-time symmetric calculation and compare the results to the reference results in sym_output.json.

Test Parameters

The params.json looks something like this:

{
  "calculation_method": "iterative_linear",
  "rtol": 1e-8,
  "atol": {
    "default": 1e-8,
    ".+_residual": 1e-4
  }
}

You need to specify the method to use for the calculation, the relative and absolute tolerance to compare the calculation results with the reference results. For rtol you always give one number. For atol you can also give one number, or you can give a dictionary with regular expressions to match the attribute names. In this way you can have fine control of individual tolerance for each attribut (e.g. active/reactive power). In the example it has an absolute tolerance of 1e-4 for attributes which ends with _residual and 1e-8 for everything else.

The calculation_method can be one string or list of strings. In the latter case, the test program will run the validation test mutilple times using all the specified methods.

See below for details.

JSON Data Format

The data format is well explained in these resources Serialization documentation and some examples of Serialization are given in Serialization notebook

Empty Result File

If you encounter a crash for a certain dataset. You can also create the input data into JSON files. In this case you might not have any reference result to compare, because you just need to find where the crash happens. You still need an empty (dictionary) result file to trigger the calculation.

For sym_output.json:

{
  "attributes": {},
  "data": {},
  "is_batch": false,
  "type": "sym_output",
  "version": "1.0"
}

For sym_output_batch.json:

{
  "attributes": {},
  "data": [{}, {}, {}],
  "is_batch": true,
  "type": "sym_output",
  "version": "1.0"
}

Helper Functions to Import and Export

In the module power_grid_model.utils we have some helper functions to import a json file to a power-grid-model compatible dataset, or the other way around.

Please refer to the documentation for detailed function signature.

In this notebook we export the example network from Power Flow to json.

# first build the network

import numpy as np
import pandas as pd

from power_grid_model import AttributeType, ComponentType, DatasetType, LoadGenType, PowerGridModel, initialize_array

# network

# node
node = initialize_array(DatasetType.input, ComponentType.node, 3)
node[AttributeType.id] = [1, 2, 6]
node[AttributeType.u_rated] = [10.5e3, 10.5e3, 10.5e3]

# line
line = initialize_array(DatasetType.input, ComponentType.line, 3)
line[AttributeType.id] = [3, 5, 8]
line[AttributeType.from_node] = [1, 2, 1]
line[AttributeType.to_node] = [2, 6, 6]
line[AttributeType.from_status] = [1, 1, 1]
line[AttributeType.to_status] = [1, 1, 1]
line[AttributeType.r1] = [0.25, 0.25, 0.25]
line[AttributeType.x1] = [0.2, 0.2, 0.2]
line[AttributeType.c1] = [10e-6, 10e-6, 10e-6]
line[AttributeType.tan1] = [0.0, 0.0, 0.0]
line[AttributeType.i_n] = [1000, 1000, 1000]

# load
sym_load = initialize_array(DatasetType.input, ComponentType.sym_load, 2)
sym_load[AttributeType.id] = [4, 7]
sym_load[AttributeType.node] = [2, 6]
sym_load[AttributeType.status] = [1, 1]
sym_load[AttributeType.type] = [LoadGenType.const_power, LoadGenType.const_power]
sym_load[AttributeType.p_specified] = [20e6, 10e6]
sym_load[AttributeType.q_specified] = [5e6, 2e6]

# source
source = initialize_array(DatasetType.input, ComponentType.source, 1)
source[AttributeType.id] = [10]
source[AttributeType.node] = [1]
source[AttributeType.status] = [1]
source[AttributeType.u_ref] = [1.0]

# all
input_data = {
    ComponentType.node: node,
    ComponentType.line: line,
    ComponentType.sym_load: sym_load,
    ComponentType.source: source,
}

Export to JSON

We can use the fuction json_serialize_to_file to export the input data to a json file.

import tempfile
from pathlib import Path

from power_grid_model.utils import json_serialize_to_file

temp_path = Path(tempfile.gettempdir())
json_serialize_to_file(temp_path / "input.json", input_data)

# we can display the json file
with (temp_path / "input.json").open("r") as f:
    print(f.read())

{
  "version": "1.0",
  "type": "input",
  "is_batch": false,
  "attributes": {},
  "data": {
    "node": [
      {"id": 1, "u_rated": 10500},
      {"id": 2, "u_rated": 10500},
      {"id": 6, "u_rated": 10500}
    ],
    "line": [
      {"id": 3, "from_node": 1, "to_node": 2, "from_status": 1, "to_status": 1, "r1": 0.25, "x1": 0.20000000000000001, "c1": 1.0000000000000001e-05, "tan1": 0, "i_n": 1000},
      {"id": 5, "from_node": 2, "to_node": 6, "from_status": 1, "to_status": 1, "r1": 0.25, "x1": 0.20000000000000001, "c1": 1.0000000000000001e-05, "tan1": 0, "i_n": 1000},
      {"id": 8, "from_node": 1, "to_node": 6, "from_status": 1, "to_status": 1, "r1": 0.25, "x1": 0.20000000000000001, "c1": 1.0000000000000001e-05, "tan1": 0, "i_n": 1000}
    ],
    "sym_load": [
      {"id": 4, "node": 2, "status": 1, "type": 0, "p_specified": 20000000, "q_specified": 5000000},
      {"id": 7, "node": 6, "status": 1, "type": 0, "p_specified": 10000000, "q_specified": 2000000}
    ],
    "source": [
      {"id": 10, "node": 1, "status": 1, "u_ref": 1}
    ]
  }
}

Import JSON

We can use the fuction json_deserialize_from_file to import the input data from a json file.

# round trip and run power flow

from power_grid_model.utils import json_deserialize_from_file

imported_data = json_deserialize_from_file(temp_path / "input.json")

pgm = PowerGridModel(imported_data)
result = pgm.calculate_power_flow()

print(pd.DataFrame(result[ComponentType.node]))

   id  energized      u_pu             u   u_angle             p             q
0   1          1  0.998988  10489.375043 -0.003039  3.121451e+07  6.991358e+06
1   2          1  0.952126   9997.325181 -0.026031 -2.000000e+07 -5.000000e+06
2   6          1  0.962096  10102.012975 -0.021895 -1.000000e+07 -2.000000e+06

Import and Export Batch Update/Result Dataset

You can use the same function to import and export batch update/result dataset for batch calculation.

# create batch set

load_profile = initialize_array(DatasetType.update, ComponentType.sym_load, (3, 2))
load_profile[AttributeType.id] = [[4, 7]]
# this is a scale of load from 0% to 100%
load_profile[AttributeType.p_specified] = [[30e6, 15e6]] * np.linspace(0, 1, 3).reshape(-1, 1)


time_series_mutation = {ComponentType.sym_load: load_profile}

# export and print

json_serialize_to_file(temp_path / "update_batch.json", time_series_mutation)

with (temp_path / "update_batch.json").open("r") as f:
    print(f.read())

{
  "version": "1.0",
  "type": "update",
  "is_batch": true,
  "attributes": {},
  "data": [
    {
      "sym_load": [
        {"id": 4, "p_specified": 0},
        {"id": 7, "p_specified": 0}
      ]
    },
    {
      "sym_load": [
        {"id": 4, "p_specified": 15000000},
        {"id": 7, "p_specified": 7500000}
      ]
    },
    {
      "sym_load": [
        {"id": 4, "p_specified": 30000000},
        {"id": 7, "p_specified": 15000000}
      ]
    }
  ]
}

# import round trip, calculate

imported_batch_update = json_deserialize_from_file(temp_path / "update_batch.json")

batch_result = pgm.calculate_power_flow(update_data=imported_batch_update)

print(batch_result[ComponentType.sym_load][AttributeType.p])

[[       0.        0.]
 [15000000.  7500000.]
 [30000000. 15000000.]]

Detailed configuration with the params.json

Validation cases with exceptions

In certain cases, you may want to create test cases that either “should raise an exception” and/or “currently have a behavior that differs from the intended one”. Examples are:

  • To create a test case for a known explicitly forbidden input, such as an unobservable grid.

    • In this case, the validation case should raise an exception.

  • To create a repro case for a bug.

    • In this case, the validation case currently has behavior that differs from the intended behavior.

  • To create a test for behavior that is not implemented yet, like a new calculation method.

    • In this case, the validation case currently has behavior that differs from the intended behavior.

  • To create a behavioral test for bad input when applying BDD (behavior driven development) practices.

    • In this case, the validation case both should raise an exception and currently has behavior that differs from the intended behavior.

To support the two use cases, two additional keywords are exposed: raises (to denote that raising is intended behavior) and xfail (to denote that the current behavior differs from the intended behavior). The values are dicts that contain a raises phrase and a reason to denote the exact intended/expected exception type, as well as a human-readable explanation about why that exception is intended/expected. Only exceptions known to the PGM tests can be added to the configuration. Note that the list of known exceptions is non-exhaustive, so it may be necessary to add a new exception type to the list of known exceptions.

In addition to the PGM exception types and some Python built-in exception types, there are two types worth explicitly mentioning:

  • AssertionError can be used to denote tests in which the actual values are different from the expected values.

  • Failed can be used to denote tests that should raise (raises) but actually do not raise any exceptions at all (xfail).

    • NOTE: If _pytest.outcomes.Failed is not found, there is a fall-back to the default pytest.mark.xfail implementation. That means, that a test is marked xfail if it throws any exception that is not the intended exception, not fail. This is a known limitation of the current implementation.

The following example shows how a test can be created intended to test future behavior.

{
  "calculation_method": "iterative_linear",
  "rtol": 1e-8,
  "atol": {
    "default": 1e-8,
    ".+_residual": 1e-4
  },
  "raises": {
    "raises": "NotObservableError",
    "reason": "The grid does not contain a voltage sensor"
  },
  "xfail": {
    "raises": "SparseMatrixError",
    "reason": "A sufficient observability check for meshed grids is not yet implemented. See also https://github.com/PowerGridModel/power-grid-model/issues/864."
  }
}

By default, the Power Grid Model test configuration accepts XFAIL cases - because they represent known issues - but rejects XPASS cases, i.e., the actual behavior was as intended, even though we expected it to be different. XPASS cases can happen because a known issue was fixed, or it can be the result of a newly introduced bug. It is up to the developer to resolve the conflict. Providing a good reason when marking something xfail can help with this decision. See https://docs.pytest.org/en/stable/reference/reference.html#pytest-mark-xfail for details.

The following table shows how the different configurations can be used. Exception types AError, BError and CError denote exceptions known to the PGM tests.

Intended behavior

Expected behavior

Keywords

Actual behavior: pass

Actually raises AError

Actually raises BError

actually raises CError

pass

pass

{}

. (PASS)

F (AError)

F (BError)

F (CError)

pass

raises AError

{"xfail": {"raises": "AError"}}

F (XPASS)

x (XFAIL)

F (BError)

F (CError)

raises AError

pass

{"raises": {"raises": "AError"}, "xfail": {"raises": "Failed"}}

x (XFAIL)

F (XPASS)

F (BError) if _pytest.outcomes.Failed is available, else x (XFAIL)

F (CError) if _pytest.outcomes.Failed is available, else x (XFAIL)

raises AError

raises AError

{"raises": {"raises": "AError"}}

F (Failed)

. (PASS)

F (BError)

F (CError)

raises AError

raises BError

{"raises": {"raises": "AError"}, "xfail": {"raises": "BError"}}

F (Failed)

F (XPASS)

x (XFAIL)

F (CError)

Calculation method-specific configuration

In rare cases, for instance when creating a new calculation method, calculation method-specific configuration may be necessary. This can be done via the extra_params keyword, which, for each overloaded calculation method, contains objects similar to the root params.json, but that is applied as a patch for that specific calculation method run. The following example illustrates that.

{
  "calculation_method": [
    "iterative_linear",
    "newton_raphson"
  ],
  "rtol": 1e-8,
  "atol": {
    "default": 1e-8,
    ".+_residual": 5e-4
  },
  "extra_params": {
    "newton_raphson": {
      "experimental_features": "enabled",
      "xfail": {
        "raises": "SparseMatrixError",
        "reason": "Current sensors are not yet implemented for this calculation method"
      }
    }
  }
}