gtopt Python Scripts

Package organization

Each command-line tool lives in its own Python package directory under scripts/:

Dependencies

Basic usage

# Auto mode (default) – picks the best aggregation for your case size
gtopt_diagram cases/ieee_9b/ieee_9b.json -o ieee9b.svg

# Interactive HTML with physics simulation
gtopt_diagram cases/ieee_9b/ieee_9b.json --format html -o ieee9b.html

# Mermaid Markdown snippet (no extra dependencies)
gtopt_diagram cases/ieee_9b/ieee_9b.json --format mermaid

# Network-only: no generator nodes (clean topology view)
gtopt_diagram large_case.json --no-generators -o topo.svg

# Planning time-structure diagram
gtopt_diagram cases/c0/system_c0.json --diagram-type planning --format html

Output formats (<tt>--format</tt>)

Aggregation modes (<tt>--aggregate</tt>)

Reducing large diagrams

# Keep only buses ≥ 220 kV (lump low-voltage buses into HV neighbours)
gtopt_diagram large_case.json --voltage-threshold 220 -o hv_topo.svg

# Show only hydro generators within 3 hops of a specific bus
gtopt_diagram large_case.json --filter-type hydro --focus-bus Chapo220 --focus-hops 3

# Hard node-count cap: escalate aggregation until ≤ 50 nodes remain
gtopt_diagram large_case.json --max-nodes 50 -o compact.svg

# Keep only the top-2 generators per bus by pmax
gtopt_diagram large_case.json --top-gens 2 -o top2.svg

Visual features

• Voltage-based line coloring: transmission lines are drawn with color intensity and width proportional to the bus voltage level (higher voltage lines appear darker and wider).
• Reservoir sizing: reservoir nodes are scaled by their storage capacity (emax), so larger reservoirs appear visually larger in the diagram.
• Reserve zones and provisions: reserve_zone_array entries are rendered as zone nodes connected to the generators that provide reserves via reserve_provision_array.
• Generator and demand profiles: generator_profile_array and demand_profile_array entries are rendered as profile nodes linked to their parent generator or demand.
• Colorblind palette: use --palette colorblind to switch all element colors to a palette designed for color-vision deficiency accessibility.

File-referenced value resolution

When fields like pmax or lmax are stored as Parquet file references, gtopt_diagram resolves them using --scenario, --stage, and --block (all default to UID 1). This controls which row of the Parquet file is used for sizing and labeling diagram elements.

All options

positional arguments:
  json_file             gtopt JSON planning file

options:
  -t, --diagram-type    topology (default) or planning
  -f, --format          dot | png | svg | pdf | mermaid | html  (default: svg)
  -o, --output          output file path
  -s, --subsystem       full | electrical | hydro  (default: full)
  -L, --layout          dot | neato | fdp | sfdp | circo | twopi
  -d, --direction       LR | TD | BT | RL  (Mermaid direction, default: LR)
  --clusters            Group in Graphviz sub-clusters
  --palette             default | colorblind  (default: default)
  --scenario UID        Scenario UID for resolving file-referenced values (default: 1)
  --stage UID           Stage UID for resolving file-referenced values (default: 1)
  --block UID           Block UID for resolving file-referenced values (default: 1)

reduction options:
  -a, --aggregate       auto | none | bus | type | global  (default: auto)
  --no-generators       Omit all generator nodes
  -g, --top-gens N      Keep only top-N generators per bus by pmax
  --filter-type TYPE    Show only: hydro solar wind thermal battery
  --focus-bus BUS       Show only elements within N hops of BUS (repeatable)
  --focus-generator GEN Focus on bus(es) connected to these generators (by name or uid)
  --focus-area KV       Focus on buses at or above this voltage level (kV)
  --focus-hops N        Hops for --focus-bus (default: 2)
  --max-nodes N         Hard cap; escalate aggregation until ≤ N nodes
  -V, --voltage-threshold KV  Lump buses below KV into HV neighbours
  --hide-isolated       Remove unconnected nodes
  --compact             Omit detail labels (names/counts only)

Hydro system conversion

When plpcnfce.dat contains reservoir (embalse) or series (serie) centrals, plp2gtopt converts the cascaded hydro system into gtopt arrays. Two additional optional PLP files extend the hydro model:

Both plpcenre.dat and plpcenfi.dat are optional — if absent, the corresponding arrays are simply not written. When present and non-empty they are silently parsed and appended to the JSON output.

**plpcenre.dat format** (Archivo de Rendimiento de Embalses):

# Number of entries
N
# For each entry:
'CENTRAL_NAME'     ← turbine (central) name
'EMBALSE_NAME'     ← reservoir name
mean_efficiency    ← fallback efficiency [MW·s/m³]
num_segments       ← number of piecewise-linear segments
idx  volume  slope  constant  scale  ← one line per segment

**plpcenfi.dat format** (Archivo de Centrales Filtración):

# Number of entries
N
# For each entry:
'CENTRAL_NAME'     ← waterway source central name
'EMBALSE_NAME'     ← receiving reservoir name
slope  constant    ← seepage model [m³/s/dam³] and [m³/s]

Basic usage

# Default: reads ./input, writes ./output.json + ./output/
plp2gtopt

# Explicit input/output directories
plp2gtopt -i plp_case_dir -o gtopt_case_dir

# Limit conversion to the first 5 stages
plp2gtopt -i input/ -s 5

# Single hydrology (1-based, default)
plp2gtopt -i input/ -y 1

# Two hydrology scenarios (1-based) with 60/40 probability split
plp2gtopt -i input/ -y 1,2 -p 0.6,0.4

# Range selector: hydrologies 1, 2, and 5 through 10
plp2gtopt -i input/ -y 1,2,5-10

# Group PLP stages 1–4 into phase 1, then one stage per phase after
plp2gtopt -i input/ --stages-phase '1:4,5,6,7,8,9,10,...'

# Apply a 10% annual discount rate
plp2gtopt -i input/ -d 0.10

# Verbose debug output
plp2gtopt -i input/ -l DEBUG

Hydrology index format (<tt>-y</tt> / <tt>--hydrologies</tt>)

Hydrology indices follow the Fortran 1-based convention: index 1 refers to the first hydrology column in the PLP data files. The argument accepts comma-separated values and ranges:

The internal 0-based index stored in the output JSON ("hydrology" field in scenario_array) is automatically computed as input_index - 1.

Phase layout (<tt>--stages-phase</tt>)

By default, phase assignment is controlled by --solver:

• sddp (default): one phase per PLP stage
• mono / monolithic: one phase covering all stages

The --stages-phase option overrides this with an explicit mapping. Tokens are comma-separated and use 1-based PLP stage indices:

# Stages 1-4 as phase 1, stages 5-10 each as their own phase,
# then one stage per phase for any remaining stages
plp2gtopt -i input/ --stages-phase '1:4,5,6,7,8,9,10,...'

# Group all stages into two phases: 1-12 and 13-24
plp2gtopt -i input/ --stages-phase '1:12,13:24'

Pasada (run-of-river) hydro modeling (<tt>--pasada-hydro</tt>)

By default (--pasada-hydro, enabled), PLP pasada (run-of-river) centrals are converted into the full gtopt hydro topology: a junction, waterway, turbine, and flow element for each central. This preserves hydrological connectivity and allows the solver to model water balance constraints.

Use --no-pasada-hydro to revert to the legacy behavior, where pasada centrals are modeled as generators with time-series profiles containing normalized capacity factors derived from the PLP afluent data.

Block-to-hour map (<tt>indhor.csv</tt>)

When the PLP input directory contains indhor.csv, plp2gtopt reads it and writes a normalised BlockHourMap/block_hour_map.parquet file alongside the other Parquet outputs. A "block_hour_map" key is added to the simulation section of the output JSON so that post-processing tools (e.g. ts2gtopt) can reconstruct hourly time-series from block-granularity solver output.

The indhor.csv format has columns: Año, Mes, Dia, Hora, Bloque (all integers; Hora is 1-based 1-24).

ZIP output (<tt>-z</tt> / <tt>--zip</tt>)

The -z flag creates a single ZIP archive that bundles the JSON configuration file and all Parquet/CSV data files together, preserving the full output directory structure. This archive is directly compatible with gtopt_guisrv (upload via the GUI) and gtopt_websrv (submit via the REST API):

plp2gtopt -z -i plp_case_2y -o gtopt_case_2y
# Produces: gtopt_case_2y.zip

The ZIP layout is:

gtopt_case_2y.zip
├── gtopt_case_2y.json          ← main system/simulation/options config
└── gtopt_case_2y/              ← input_directory (data files)
    ├── Demand/
    │   └── lmax.parquet
    ├── Generator/
    │   ├── pmin.parquet
    │   └── pmax.parquet
    └── Afluent/
        └── afluent.parquet

Note: the input_directory field in the JSON options matches the name of the subdirectory inside the ZIP, so gtopt_guisrv and gtopt_websrv can locate the data files without any extra configuration.

Conversion statistics

After a successful conversion, plp2gtopt logs statistics (at INFO level) similar to the pre-solve statistics printed by the gtopt solver:

=== System statistics ===
  System name     : plp2gtopt
=== System elements  ===
  Buses           : 2
  Generators      : 3
  Generator profs : 1
  Demands         : 2
  Lines           : 1
  Batteries       : 0
  Converters      : 0
  Junctions       : 2
  Waterways       : 1
  Reservoirs      : 1
  Turbines        : 1
=== Simulation statistics ===
  Blocks          : 8760
  Stages          : 5
  Scenarios       : 2
=== Key options ===
  use_single_bus  : False
  scale_objective : 1000
  demand_fail_cost: 1000
  input_directory : gtopt_case_2y
  annual_discount : 0.1
=== Conversion time ===
  Elapsed         : 1.234s

Use -l DEBUG to also see which individual .dat files are being parsed.

All options

Isolated centrals

During hydro topology conversion, PLP centrals that have no bus assignment (bus <= 0), no waterway connections, and are not referenced by any other central are considered isolated and are silently skipped. After the conversion statistics, a "Skipped Centrals" section lists all isolated centrals by name so the user can verify that they are genuinely unused.

Error messages

plp2gtopt raises descriptive errors for common problems:

Basic usage

# Convert the default IEEE 30-bus built-in network → ieee30b.json
pp2gtopt

# Convert a saved pandapower JSON file
pp2gtopt -f my_network.json -o my_case.json

# Convert a MATPOWER case file
pp2gtopt -f case39.m -o case39.json

# Convert a pandapower Excel workbook
pp2gtopt -f network.xlsx -o network.json

# Use a specific built-in test network
pp2gtopt -n case14 -o ieee14b.json

# List all available built-in test networks
pp2gtopt --list-networks

Input from file (<tt>-f / --file</tt>)

-f FILE loads any pandapower network saved to disk. The format is auto-detected from the file extension:

The output JSON system name is derived from the file stem (e.g. case39.m → "case39").

Available built-in networks (<tt>-n / --network</tt>)

All options

Basic usage

# Basic conversion – output JSON and input directory derived from workbook name
igtopt system.xlsx

# Write output to an explicit JSON file
igtopt system.xlsx -j output/system.json

# Pretty-printed JSON (4-space indented), skip null/NaN values
igtopt system.xlsx --pretty --skip-nulls

# CSV time-series files instead of Parquet
igtopt system.xlsx -f csv

# Bundle JSON + data files into a ZIP archive (ready for gtopt_guisrv/websrv)
igtopt system.xlsx --zip

# Convert multiple workbooks in one run (merges into a single JSON)
igtopt case_a.xlsx case_b.xlsx -d /data/input

# Validate the workbook without writing output (exit 0 = OK, 1 = errors)
igtopt system.xlsx --validate

# Proceed even if some sheets have errors (partial output)
igtopt system.xlsx --ignore-errors

# Show debug log messages
igtopt system.xlsx -l DEBUG

Template generation (<tt>-T</tt> / <tt>--make-template</tt>)

The --make-template flag reads the gtopt C++ JSON header files (include/gtopt/json/) and generates a ready-to-use Excel template. Re-running after adding a new JSON element to the C++ source automatically produces an up-to-date workbook.

# Regenerate docs/templates/gtopt_template.xlsx (default output)
igtopt --make-template

# Write to a custom path (reuses the -j/--json-file flag)
igtopt --make-template -j /tmp/my_template.xlsx

# Print the sheet list that would be generated (no file written)
igtopt --make-template --list-sheets

# Custom C++ header directory
igtopt --make-template --header-dir /path/to/include/gtopt

ZIP output (<tt>-z</tt> / <tt>--zip</tt>)

The -z flag creates a single ZIP archive that bundles the JSON configuration file and all Parquet/CSV data files together, preserving the full directory structure. This archive is directly compatible with gtopt_guisrv (upload via the GUI) and gtopt_websrv (submit via the REST API):

igtopt system.xlsx --zip
# Produces: system.zip
#   system.zip
#   ├── system.json
#   └── system/
#       ├── Demand/
#       │   └── lmax.parquet
#       └── GeneratorProfile/
#           └── profile.parquet

Conversion statistics

After a successful conversion, igtopt logs statistics (at INFO level) similar to those printed by plp2gtopt:

=== System statistics ===
  Buses           : 57
  Generators      : 7
  Demands         : 42
  Lines           : 80
=== Simulation statistics ===
  Blocks          : 1
  Stages          : 1
  Scenarios       : 1
=== Key options ===
  use_single_bus  : False
  scale_objective : 1000
  demand_fail_cost: 1000
  input_directory : system
=== Conversion time ===
  Elapsed         : 0.123s

Excel workbook format

The workbook can contain any of the following named sheets. Sheets whose name starts with . (e.g. .notes) are silently skipped.

<component_type>@<field_name>

All options

Concepts

Basic usage

# Project a single hourly CSV onto an auto-generated 12-stage × 24-block horizon
ts2gtopt demand.csv -y 2023 -o input/

# Write 4 seasonal stages × 6 blocks each
ts2gtopt solar.csv -y 2023 --stages 4 --blocks 6 -o input/

# Use a custom horizon JSON and export updated block durations
ts2gtopt pmax.csv -y 2023 -H my_horizon.json --output-horizon updated_horizon.json -o input/

# Use the horizon embedded in an existing planning JSON (reads simulation.stage_array / block_array)
ts2gtopt load.csv -y 2023 -P case.json -o input/

Reconstructing hourly output

After running the gtopt solver (which produces output/Generator/generation_sol.csv etc.), use the hour_block_map embedded in the planning JSON to expand block-level results back into a full hourly time-series:

from ts2gtopt import write_output_hours

# Reads hour_block_map from case.json, writes output_hour/ next to output/
written = write_output_hours("bat_4b_2023.json")
# → output_hour/Generator/generation_sol.csv
#    scenario  hour   uid:1   uid:2
#    1          0    127.5    0.0
#    1          1    124.9    0.0
#    ...  (8760 rows for a full-year case)

Or from the public API:

from ts2gtopt import build_hour_block_map, reconstruct_output_hours, load_horizon

h = load_horizon("my_horizon.json")
hour_map = build_hour_block_map(h, year=2023)
reconstruct_output_hours("output/", hour_map, output_hour_dir="output_hour/")

CLI reference

ts2gtopt [options] INPUT [INPUT ...]

Positional arguments:
  INPUT                  Input time-series file(s) (CSV or Parquet)

Options:
  -y, --year YEAR        Calendar year for the projection (required unless -H/-P is given with explicit dates)
  -o, --output DIR       Output directory for block schedule files (default: current directory)
  -f, --format {parquet,csv}
                         Output file format (default: parquet)
  -s, --stages N         Number of planning stages (default: 12, one per calendar month)
  -b, --blocks N         Number of representative blocks per stage (default: 24, one per hour of day)
  -H, --horizon FILE     Load planning horizon from a JSON file
  -P, --planning FILE    Load horizon from an existing gtopt planning JSON
  --output-horizon FILE  Write the duration-updated horizon to FILE after projecting
  --agg {mean,median,min,max,sum}
                         Aggregation function (default: mean)
  --interval-hours H     Duration of each input observation in hours (auto-detected by default)
  --verify               Print energy-conservation ratios after projection
  -v, --verbose          Enable verbose logging

Basic usage

# Convert a gtopt case to pandapower JSON
gtopt2pp cases/ieee_9b_ori/ieee_9b_ori.json

# Specify output file
gtopt2pp cases/ieee_9b_ori/ieee_9b_ori.json -o ieee9b_pp.json

# Convert and solve DC OPF
gtopt2pp cases/ieee_9b_ori/ieee_9b_ori.json --solve

# Convert all blocks (one output file per block)
gtopt2pp cases/ieee_9b/ieee_9b.json --all-blocks

# Select specific blocks (comma-separated, ranges)
gtopt2pp cases/ieee_9b/ieee_9b.json -b 1,5-10

# Run pandapower diagnostic on converted network
gtopt2pp cases/ieee_14b_ori/ieee_14b_ori.json --diagnostic

Multi-block support

When the source case has multiple blocks (e.g. 24-hour dispatch), use --all-blocks or -b SPEC to produce one pandapower network per block:

gtopt2pp cases/ieee_9b/ieee_9b.json --all-blocks
# Produces: ieee_9b_pp_b1.json, ieee_9b_pp_b2.json, …, ieee_9b_pp_b24.json

Elements skipped during conversion

The following gtopt elements have no pandapower equivalent and are silently skipped: batteries, converters, junctions, waterways, reservoirs, turbines, filtrations, flows. A summary of skipped elements is logged.

All options

Basic usage

# Auto-detect case type from CWD
run_gtopt

# PLP case: auto-convert to gtopt and solve
run_gtopt plp_case_2y

# gtopt case directory: solve directly
run_gtopt cases/ieee_9b

# Explicit JSON file
run_gtopt cases/ieee_9b/ieee_9b.json

# Pass extra arguments to the gtopt binary after --
run_gtopt cases/ieee_9b -- --set use_single_bus=true --stats

Pre-flight checks (enabled by default)

Before invoking the solver, run_gtopt validates:

• JSON syntax and file readability
• Input file existence (Parquet/CSV references)
• Output directory writability
• Compression codec availability (auto-fallback if unavailable)
• System/simulation statistics (gtopt_check_json --info)
• Full JSON validation (gtopt_check_json)

Disable with --no-check or abort on any warning with --strict.

Post-flight checks

After the solver completes, run_gtopt automatically:

• Analyzes any error LP files via gtopt_check_lp
• Validates solver output via gtopt_check_output
• Prints a solution summary

All options

Basic usage

# Validate a JSON case and report issues
gtopt_check_json cases/ieee_9b/ieee_9b.json

# Print system/simulation statistics only (no validation)
gtopt_check_json --info cases/ieee_9b/ieee_9b.json

# Show detailed simulation structure (scenarios, stages, phases, apertures)
gtopt_check_json --show-simulation cases/sddp_hydro_3phase/sddp_hydro_3phase.json

# Validate multiple JSON files (merged before checking)
gtopt_check_json system.json overrides.json

# Run interactive configuration setup
gtopt_check_json --init-config

Validation checks

The tool runs configurable checks organized by severity:

Individual checks can be enabled/disabled via --init-config or the configuration file (~/.gtopt.conf).

Exit codes

All options

Basic usage

# Analyze an LP file
gtopt_check_lp error_0.lp

# Analyze a gzip-compressed LP file
gtopt_check_lp error_0.lp.gz

# Auto-find the newest error*.lp[.gz] in the current directory
gtopt_check_lp --last

# Static analysis only (no solver invocation)
gtopt_check_lp error_0.lp --analyze-only

# Use a specific solver
gtopt_check_lp error_0.lp --solver coinor

# Submit to NEOS remote server (requires email)
gtopt_check_lp error_0.lp --solver neos --email user@example.com

# Run interactive configuration setup
gtopt_check_lp --init-config

Analysis pipeline

1. Static analysis — parses the LP file directly:
   • Variables with conflicting bounds (lb > ub)
   • Empty or fixed constraints
   • Numerical range issues (very large/small coefficients)
   • Duplicate constraint names
   • Problem statistics (rows, columns, non-zeros)
2. Local IIS finding — tries available solvers in order: CPLEX → HiGHS → COIN-OR CLP → CBC → GLPK
3. NEOS remote analysis — submits to https://neos-server.org via XML-RPC using CPLEX (requires email)
4. AI diagnostics *(optional)* — expert infeasibility diagnosis from Claude, OpenAI, DeepSeek, or GitHub AI

Quiet mode

When called with --quiet (used automatically by the gtopt binary and run_gtopt), the tool never fails (always exits 0), never prompts for input, and handles all errors gracefully with warnings.

All options

Basic usage

# Analyze output in a case directory
gtopt_check_output cases/ieee_9b

# Explicit paths to results and JSON
gtopt_check_output -r output/ -j ieee_9b.json

# Quiet mode — only show warnings and critical findings
gtopt_check_output cases/ieee_9b --quiet

# Run interactive configuration setup
gtopt_check_output --init-config

Checks performed

• Output completeness — verifies expected output files exist (solution.csv, generation_sol, balance_dual, etc.)
• Load shedding analysis — detects unserved energy in fail_sol
• Generation/demand balance — validates energy conservation
• Line congestion ranking — identifies most congested transmission lines
• LMP statistics — locational marginal price analysis
• Cost breakdown — dispatches by generator cost tier

All options

Basic usage

# List all available LP solver plugins
gtopt_check_solvers --list

# Check all available solvers (default)
gtopt_check_solvers

# Check a specific solver
gtopt_check_solvers --solver clp

# Check multiple specific solvers
gtopt_check_solvers --solver clp --solver highs

# Use an explicit gtopt binary path
gtopt_check_solvers --gtopt-bin /opt/gtopt/bin/gtopt

# Run only a subset of built-in tests
gtopt_check_solvers --test single_bus_lp --test kirchhoff_lp

# Verbose output (show failure details)
gtopt_check_solvers --verbose

Built-in test cases

For each test the tool:

1. Writes the problem as a temporary gtopt JSON file
2. Calls gtopt <file> --solver <name> with a per-test timeout
3. Reads output/solution.csv and validates status and obj_value

Exit codes

All options

Binary discovery

The tool searches for the gtopt binary in this order:

1. GTOPT_BIN environment variable
2. gtopt on PATH
3. Standard build directories (build/standalone/gtopt, etc.)

Basic usage

# Compress an LP file
gtopt_compress_lp file.lp

# Use a specific compression codec
gtopt_compress_lp file.lp --codec zstd

# Quiet mode (non-interactive, never fails)
gtopt_compress_lp file.lp --quiet

# Show available compression tools
gtopt_compress_lp --list-tools

# Run interactive setup
gtopt_compress_lp --init-config

Supported compressors

Compression cascade

1. Codec hint from --codec (if provided and available)
2. Configured compressor from config file (auto → first available)
3. First available tool found on PATH
4. Skip compression (leave original unchanged)

All options

Configuration file

Default location: ~/.gtopt.conf. The file uses INI format with a [global] section for shared settings, per-tool sections, and a [gtopt] section for the C++ binary:

[global]
ai_enabled  = false
ai_provider = claude
ai_model    =
color       = auto

[gtopt]
solver              = highs
algorithm           = barrier
threads             = 4
output-format       = parquet
output-compression  = zstd

[gtopt_check_json]
check_uid_uniqueness = true

[gtopt_check_output]
congestion_top_n = 10

[run_gtopt]
default_threads = 0

The [gtopt] section provides default values for the C++ binary's command-line options. CLI flags always take precedence. See Usage Guide for the full list of supported keys.

Initializing configuration

Each tool that uses gtopt_config provides an --init-config flag that runs an interactive setup wizard to create or update its section:

gtopt_check_json --init-config
gtopt_check_lp --init-config
gtopt_check_output --init-config
run_gtopt --init-config     # via run_gtopt --list-checks / --init-config

Text mode output

In headless mode (--no-gui), the tool prints a summary table to stdout:

[Time]  [Iter]  [LB]           [UB]           [Gap]      [Status]
  7.1s   45     12345.1234   12450.5678     0.008765   running

All options

gtopt_guisrv (browser GUI)

1. Start the GUI service:

   gtopt_guisrv
   # Open http://localhost:5001

2. Click Upload case and select the .zip file produced by plp2gtopt -z.
3. Edit the case if needed, then click Solve to submit it to the webservice.

For installation and service setup, see guiservice/INSTALL.md.

gtopt_websrv (REST API)

Submit a ZIP directly to the REST API:

curl -X POST http://localhost:3000/api/solve \
  -F "file=@gtopt_case_2y.zip"

The server runs the gtopt solver and returns a results ZIP containing the solution files.

For full API reference and deployment instructions, see webservice/INSTALL.md.