Updating to PSY5 and `StorageSystemsSimulations` v0.14 cause `DimensionMismatch` error.

[yasirroni]

Updating to PSY5 and StorageSystemsSimulations v0.14 cause DimensionMismatch error.

julia> using Pkg

julia> println("PowerSystems.jl version: ", Pkg.installed()["PowerSystems"])
┌ Warning: Pkg.installed() is deprecated
└ @ Pkg ~/.julia/juliaup/julia-1.11.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.11/Pkg/src/Pkg.jl:785
PowerSystems.jl version: 5.3.0

julia> println("PowerSimulations.jl version: ", Pkg.installed()["PowerSimulations"])
┌ Warning: Pkg.installed() is deprecated
└ @ Pkg ~/.julia/juliaup/julia-1.11.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.11/Pkg/src/Pkg.jl:785
PowerSimulations.jl version: 0.32.2

julia> println("StorageSystemsSimulations.jl version: ", Pkg.installed()["StorageSystemsSimulations"])
┌ Warning: Pkg.installed() is deprecated
└ @ Pkg ~/.julia/juliaup/julia-1.11.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.11/Pkg/src/Pkg.jl:785
StorageSystemsSimulations.jl version: 0.14.0

┌ Error: Decision Problem solve failed
│   exception =
│    DimensionMismatch: column :DateTime has length 1 and column :name has length 50
│    Stacktrace:
│      [1] DataFrames.DataFrame(columns::Vector{Any}, colindex::DataFrames.Index; copycols::Bool)
│        @ DataFrames ~/.julia/packages/DataFrames/b4w9K/src/dataframe/dataframe.jl:212
│      [2] DataFrame
│        @ ~/.julia/packages/DataFrames/b4w9K/src/dataframe/dataframe.jl:193 [inlined]
│      [3] DataFrames.DataFrame(::Pair{Symbol, Vector{Int64}}, ::Vararg{Pair{Symbol}}; makeunique::Bool, copycols::Bool)
│        @ DataFrames ~/.julia/packages/DataFrames/b4w9K/src/dataframe/dataframe.jl:261
│      [4] DataFrame
│        @ ~/.julia/packages/DataFrames/b4w9K/src/dataframe/dataframe.jl:257 [inlined]
│      [5] to_results_dataframe(array::JuMP.Containers.DenseAxisArray{Float64, 1, Tuple{Vector{String}}, Tuple{JuMP.Containers._AxisLookup{Dict{String, Int64}}}}, timestamps::Nothing, ::Val{TableFormat.LONG = 0})
│        @ PowerSimulations ~/.julia/packages/PowerSimulations/i092P/src/utils/jump_utils.jl:255
│      [6] _read_results(model::DecisionModel{GenericOpProblem}, key::InfrastructureSystems.Optimization.VariableKey{StorageSystemsSimulations.StorageEnergySurplusVariable, PowerSystems.EnergyReservoirStorage})
│        @ PowerSimulations ~/.julia/packages/PowerSimulations/i092P/src/operation/operation_model_interface.jl:344
│      [7] read_variable
│        @ ~/.julia/packages/PowerSimulations/i092P/src/operation/operation_model_interface.jl:339 [inlined]
│      [8] (::PowerSimulations.var"#116#121"{DecisionModel{GenericOpProblem}})(x::InfrastructureSystems.Optimization.VariableKey{StorageSystemsSimulations.StorageEnergySurplusVariable, PowerSystems.EnergyReservoirStorage})
│        @ PowerSimulations ./none:0
│      [9] grow_to!(dest::Dict{InfrastructureSystems.Optimization.VariableKey, DataFrames.DataFrame}, itr::Base.Generator{Vector{InfrastructureSystems.Optimization.VariableKey}, PowerSimulations.var"#116#121"{DecisionModel{GenericOpProblem}}}, st::Int64)
│        @ Base ./abstractdict.jl:0
│     [10] grow_to!(dest::Dict{InfrastructureSystems.Optimization.VariableKey{ReservationVariable, PowerSystems.EnergyReservoirStorage}, DataFrames.DataFrame}, itr::Base.Generator{Vector{InfrastructureSystems.Optimization.VariableKey}, PowerSimulations.var"#116#121"{DecisionModel{GenericOpProblem}}}, st::Int64)
│        @ Base ./abstractdict.jl:627
│     [11] grow_to!(dest::Dict{Any, Any}, itr::Base.Generator{Vector{InfrastructureSystems.Optimization.VariableKey}, PowerSimulations.var"#116#121"{DecisionModel{GenericOpProblem}}})
│        @ Base ./abstractdict.jl:606
│     [12] dict_with_eltype
│        @ ./abstractdict.jl:645 [inlined]
│     [13] Dict(kv::Base.Generator{Vector{InfrastructureSystems.Optimization.VariableKey}, PowerSimulations.var"#116#121"{DecisionModel{GenericOpProblem}}})
│        @ Base ./dict.jl:117
│     [14] OptimizationProblemResults(model::DecisionModel{GenericOpProblem})
│        @ PowerSimulations ~/.julia/packages/PowerSimulations/i092P/src/operation/problem_results.jl:20
│     [15] (::PowerSimulations.var"#97#98"{Bool, Bool, DecisionModel{GenericOpProblem}, Nothing})()
│        @ PowerSimulations ~/.julia/packages/PowerSimulations/i092P/src/operation/decision_model.jl:516
│     [16] with_logstate(f::PowerSimulations.var"#97#98"{Bool, Bool, DecisionModel{GenericOpProblem}, Nothing}, logstate::Base.CoreLogging.LogState)
│        @ Base.CoreLogging ./logging/logging.jl:522
│     [17] with_logger
│        @ ./logging/logging.jl:632 [inlined]
│     [18] solve!(model::DecisionModel{GenericOpProblem}; export_problem_results::Bool, console_level::Base.CoreLogging.LogLevel, file_level::Base.CoreLogging.LogLevel, disable_timer_outputs::Bool, export_optimization_problem::Bool, kwargs::@Kwargs{})
│        @ PowerSimulations ~/.julia/packages/PowerSimulations/i092P/src/operation/decision_model.jl:490
│     [19] solve!
│        @ ~/.julia/packages/PowerSimulations/i092P/src/operation/decision_model.jl:461 [inlined]
└ @ PowerSimulations ~/.julia/packages/PowerSimulations/i092P/src/operation/decision_model.jl:522
ERROR: problem was not solved successfully: InfrastructureSystems.Simulation.RunStatusModule.RunStatus.FAILED = 2
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] OptimizationProblemResults(model::DecisionModel{GenericOpProblem})
   @ PowerSimulations ~/.julia/packages/PowerSimulations/i092P/src/operation/problem_results.jl:6

[jd-lara]

I need a MWE to reproduce this.

[yasirroni]

Hi @jd-lara, I finally found a way to reproduce it using RTS data. Adding new battery cause this same error.

## Original Battery Single Step Problem
using InfrastructureSystems
using PowerSystems
using PowerSimulations
using StorageSystemsSimulations
using PowerSystemCaseBuilder
using HiGHS # solver
using Dates
using TimeSeries
using Logging
using DataFrames

const IS = InfrastructureSystems
const PS = PowerSystems
const PSS = PowerSimulations


sys = build_system(
    PSITestSystems,
    "c_sys5_bat_ems";
    add_single_time_series=true,
    add_reserves=false,
)

# Remove all time series from all batteries
for storage in get_components(EnergyReservoirStorage, sys)
    for ts_metadata in get_time_series_multiple(storage)
        remove_time_series!(sys, typeof(ts_metadata), storage, get_name(ts_metadata))
    end
end

# Create a new battery with modified name and bus
original_battery = first(get_components(EnergyReservoirStorage, sys))
new_battery = EnergyReservoirStorage(
    name = "Bat3",
    available = get_available(original_battery),
    bus = get_bus(original_battery),
    prime_mover_type = get_prime_mover_type(original_battery),
    storage_technology_type = get_storage_technology_type(original_battery),
    storage_capacity = get_storage_capacity(original_battery),
    storage_level_limits = get_storage_level_limits(original_battery),
    initial_storage_capacity_level = get_initial_storage_capacity_level(original_battery),
    rating = get_rating(original_battery),
    active_power = get_active_power(original_battery),
    input_active_power_limits = get_input_active_power_limits(original_battery),
    output_active_power_limits = get_output_active_power_limits(original_battery),
    efficiency = get_efficiency(original_battery),
    reactive_power = get_reactive_power(original_battery),
    reactive_power_limits = get_reactive_power_limits(original_battery),
    base_power = get_base_power(original_battery),
    operation_cost = get_operation_cost(original_battery),
    conversion_factor = get_conversion_factor(original_battery),
    storage_target = get_storage_target(original_battery),
    cycle_limits = get_cycle_limits(original_battery),
    services = get_services(original_battery),
    dynamic_injector = get_dynamic_injector(original_battery),
)
add_component!(sys, new_battery)

# Forecast Summary
# ┌───────────────────┬────────────────┬──────────────────┬──────────────────┬─────────────────────┬──────────────────────┬───────┬──────────────────────┬──────────────────────┬──────────────┐
# │ owner_type        │ owner_category │ name             │ time_series_type │ initial_timestamp   │ resolution           │ count │ horizon              │ interval             │ window_count │
# │ String            │ String         │ String           │ String           │ String              │ Dates.CompoundPeriod │ Int64 │ Dates.CompoundPeriod │ Dates.CompoundPeriod │ Int64        │
# ├───────────────────┼────────────────┼──────────────────┼──────────────────┼─────────────────────┼──────────────────────┼───────┼──────────────────────┼──────────────────────┼──────────────┤
# │ PowerLoad         │ Component      │ max_active_power │ Deterministic    │ 2024-01-01T00:00:00 │ 1 hour               │ 3     │ 1 day                │ 1 day                │ 2            │
# │ RenewableDispatch │ Component      │ max_active_power │ Deterministic    │ 2024-01-01T00:00:00 │ 1 hour               │ 3     │ 1 day                │ 1 day                │ 2            │
# └───────────────────┴────────────────┴──────────────────┴──────────────────┴─────────────────────┴──────────────────────┴───────┴──────────────────────┴──────────────────────┴──────────────┘

template_uc = ProblemTemplate()
set_device_model!(template_uc, Line, StaticBranch)
set_device_model!(template_uc, ThermalStandard, ThermalStandardUnitCommitment)
set_device_model!(template_uc, RenewableDispatch, RenewableFullDispatch)
set_device_model!(template_uc, PowerLoad, StaticPowerLoad)

storage_model = DeviceModel(
    EnergyReservoirStorage,
    StorageDispatchWithReserves;
    attributes=Dict(
        "reservation" => true,
        "energy_target" => true,
        "cycling_limits" => false,
        "regularization" => false,
    ),
    use_slacks=false,
)
set_device_model!(template_uc, storage_model)
set_network_model!(
    template_uc,
    NetworkModel(
        NFAPowerModel,
        use_slacks = true,
    ),
)

solver = optimizer_with_attributes(HiGHS.Optimizer, "mip_rel_gap" => 0.01)

# Build and solve single step problem
decision_model = DecisionModel(template_uc, sys; optimizer=solver, name="UC")
build!(decision_model; output_dir=mktempdir())
solve!(decision_model)
res = OptimizationProblemResults(decision_model)

[yasirroni]

I think the bug is due to the new data format that PSY4 use (time vs components) to PSY5 (rows of key value pairs).

[yasirroni]

    res_variables = read_variables(res)
    power_out = res_variables["ActivePowerOutVariable__EnergyReservoirStorage"]

PSY4

julia> power_out = res_variables["ActivePowerOutVariable__EnergyReservoirStorage"]
48×3 DataFrame
 Row │ DateTime             Bat2     Bat3    
     │ DateTime             Float64  Float64 
─────┼───────────────────────────────────────
   1 │ 2024-01-03T00:00:00      0.0      0.0
   2 │ 2024-01-03T01:00:00      0.0      0.0
   3 │ 2024-01-03T02:00:00    200.0    200.0
   4 │ 2024-01-03T03:00:00      0.0      0.0
   5 │ 2024-01-03T04:00:00      0.0      0.0
   6 │ 2024-01-03T05:00:00     41.0      0.0
   7 │ 2024-01-03T06:00:00      0.0      0.0
   8 │ 2024-01-03T07:00:00      0.0      0.0
   9 │ 2024-01-03T08:00:00      0.0      0.0
  ⋮  │          ⋮              ⋮        ⋮
  41 │ 2024-01-04T16:00:00      0.0      0.0
  42 │ 2024-01-04T17:00:00      0.0      0.0
  43 │ 2024-01-04T18:00:00      0.0      0.0
  44 │ 2024-01-04T19:00:00      0.0      0.0
  45 │ 2024-01-04T20:00:00      0.0      0.0
  46 │ 2024-01-04T21:00:00      0.0      0.0
  47 │ 2024-01-04T22:00:00      0.0      0.0
  48 │ 2024-01-04T23:00:00     -0.0      0.0
                              31 rows omitted

PSY5

julia> power_out = res_variables["ActivePowerOutVariable__EnergyReservoirStorage"]
24×3 DataFrame
 Row │ DateTime             name    value    
     │ DateTime             String  Float64  
─────┼───────────────────────────────────────
   1 │ 2024-01-01T00:00:00  Bat2      0.0
   2 │ 2024-01-01T01:00:00  Bat2      0.0
   3 │ 2024-01-01T02:00:00  Bat2      0.0
   4 │ 2024-01-01T03:00:00  Bat2      0.0
   5 │ 2024-01-01T04:00:00  Bat2      0.0
   6 │ 2024-01-01T05:00:00  Bat2      0.0
   7 │ 2024-01-01T06:00:00  Bat2      0.0
   8 │ 2024-01-01T07:00:00  Bat2      0.0
   9 │ 2024-01-01T08:00:00  Bat2      0.0
  ⋮  │          ⋮             ⋮        ⋮
  17 │ 2024-01-01T16:00:00  Bat2      0.0
  18 │ 2024-01-01T17:00:00  Bat2    143.599
  19 │ 2024-01-01T18:00:00  Bat2    135.56
  20 │ 2024-01-01T19:00:00  Bat2    141.073
  21 │ 2024-01-01T20:00:00  Bat2    113.475
  22 │ 2024-01-01T21:00:00  Bat2     64.3008
  23 │ 2024-01-01T22:00:00  Bat2     22.9923
  24 │ 2024-01-01T23:00:00  Bat2      0.0
                               7 rows omitted