Skip to content

Parse site data inputs to Python native formats #561

@Sidrcs

Description

@Sidrcs

reV version checks

  • I have confirmed this bug has not already been reported.

  • I have confirmed this bug exists on the latest version of reV.

  • I have confirmed this bug exists on the main branch of reV.

Bug Description

When the reV submits site data to reVSAMWrapper, reV parses the site data into a dictionary and submits them. However, SAM expects certain forms of site data to be in Python native formats. For instance, in pvwattv8 module, if the user wants to pass the site data for shading group. Let us consider shading_azal, SAM expects a type sequence[sequence] - nested list. However, when the site data is saved as a CSV, nested lists are converted to object and even when reV loads them, it will be submitted to SAM as an object type. In the site data, the values are to be parsed before submitting to the SAM. Also, when the site data is loaded as a dataframe and parsed as a list, reV also raises an error. The full traceback is shared. I could find a conclusive evidence or specific line numbers where reV breaks with nested tuple.

Reproducible Example

reV generation -c config_gen.json

site_data.csv
sam.json

Full Traceback

concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/SAM.py", line 418, in __setitem__
    setattr(getattr(self.pysam, group), key, value)
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: Error (-4) converting nested tuple 0 into row in matrix.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/projects/rev/environments/revruns/lib/python3.13/concurrent/futures/process.py", line 254, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/generation.py", line 746, in _run_single_worker
    raise e
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/generation.py", line 732, in _run_single_worker
    out = cls.OPTIONS[tech].reV_run(
        points_control,
    ...<6 lines>...
        bias_correct=bias_correct,
    )
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/generation.py", line 580, in reV_run
    sim.run_gen_and_econ()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/generation.py", line 425, in run_gen_and_econ
    self.run()
    ~~~~~~~~^^
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/generation.py", line 449, in run
    self.assign_inputs()
    ~~~~~~~~~~~~~~~~~~^^
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/SAM.py", line 933, in assign_inputs
    super().assign_inputs(copy.deepcopy(self.sam_sys_inputs))
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/SAM.py", line 610, in assign_inputs
    self[k] = v
    ~~~~^^^
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/SAM.py", line 429, in __setitem__
    raise SAMInputError(msg) from e
reV.utilities.exceptions.SAMInputError: Could not set input key "shading_azal" to group "Shading" in "<Pvwattsv8 object at 0x7f67633d0ed0>".
Data is: site_data (<class 'str'>)
Received the following error: "Error (-4) converting nested tuple 0 into row in matrix."
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
    from gaps.cli.config import run_with_status_updates; from reV.generation.generation import Gen; su_args = "/kfs2/projects/pxs/erosenli/conus/rev", "generation", "rev_generation"; run_with_status_updates(   Gen, {"analysis_years": [2022], "technology": "pvwattsv8", "output_request": ["cf_profile", "cf_profile_ac", "cf_mean", "cf_mean_ac", "ghi_mean", "lcoe_fcr", "ac", "dc", "clipped_power", "capital_cost", "fixed_operating_cost", "system_capacity", "system_capacity_ac", "fixed_charge_rate", "variable_operating_cost", "dc_ac_ratio"], "project_points": "/kfs2/projects/pxs/erosenli/conus/rev/project_points_inclusion_90m_13454.csv", "sam_files": {"default": "/kfs2/projects/pxs/erosenli/conus/rev/sam.json"}, "resource_file": "/datasets/NSRDB/conus/nsrdb_conus_*_2022.h5", "site_data": "/kfs2/projects/pxs/erosenli/conus/rev/site_data.csv", "gid_map": "/kfs2/projects/pxs/erosenli/conus/rev/gid_map.csv", "low_res_resource_file": None, "project_points_split_range": [0, 13454], "tag": "", "command_name": "generation", "pipeline_step": "generation", "config_file": "/kfs2/projects/pxs/erosenli/conus/rev/config_gen.json", "project_dir": "/kfs2/projects/pxs/erosenli/conus/rev", "job_name": "rev_generation", "out_dir": "/kfs2/projects/pxs/erosenli/conus/rev", "out_fpath": "/kfs2/projects/pxs/erosenli/conus/rev/rev_generation", "run_method": "run", "max_workers": None, "sites_per_worker": 100, "memory_utilization_limit": 0.8}, {"name": "rev_generation", "log_directory": "/kfs2/projects/pxs/erosenli/conus/rev/logs", "verbose": False, "node": True}, su_args,    ["project_points"])

File "/projects/rev/environments/revruns/lib/python3.13/site-packages/gaps/cli/config.py", line 602, in run_with_status_updates
    out = func(**run_kwargs)
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/generation.py", line 1146, in run
    raise e
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/generation.py", line 1137, in run
    self._parallel_run(
        max_workers=max_workers,
    ...<2 lines>...
        **kwargs,
    )
    ^
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/base.py", line 1361, in _parallel_run
    result = future.result(timeout=timeout)
  File "/projects/rev/environments/revruns/lib/python3.13/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/projects/rev/environments/revruns/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
reV.utilities.exceptions.SAMInputError: Could not set input key "shading_azal" to group "Shading" in "<Pvwattsv8 object at 0x7f67633d0ed0>".
Data is: site_data (<class 'str'>)
Received the following error: "Error (-4) converting nested tuple 0 into row in matrix.

WHEN SITE DATA IS PASSED AS NESTED LIST IN A DATAFRAME, IT STILL RAISES ERROR:

Traceback (most recent call last):
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/SAM.py", line 418, in __setitem__
    setattr(getattr(self.pysam, group), key, value)
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: Value must be numeric

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/kfs2/projects/pxs/erosenli/conus/rev/gen.py", line 60, in <module>
    main()
    ~~~~^^
  File "/kfs2/projects/pxs/erosenli/conus/rev/gen.py", line 56, in main
    skyview_gen.run(out_fpath=out_fpath.as_posix())
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/generation.py", line 1146, in run
    raise e
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/generation.py", line 1123, in run
    self.out = self._run_single_worker(pc_sub, **kwargs)
               ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/generation.py", line 746, in _run_single_worker
    raise e
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/generation.py", line 732, in _run_single_worker
    out = cls.OPTIONS[tech].reV_run(
        points_control,
    ...<6 lines>...
        bias_correct=bias_correct,
    )
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/generation.py", line 580, in reV_run
    sim.run_gen_and_econ()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/generation.py", line 425, in run_gen_and_econ
    self.run()
    ~~~~~~~~^^
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/generation.py", line 449, in run
    self.assign_inputs()
    ~~~~~~~~~~~~~~~~~~^^
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/SAM.py", line 933, in assign_inputs
    super().assign_inputs(copy.deepcopy(self.sam_sys_inputs))
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/SAM.py", line 610, in assign_inputs
    self[k] = v
    ~~~~^^^
  File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/SAM.py", line 429, in __setitem__
    raise SAMInputError(msg) from e
reV.utilities.exceptions.SAMInputError: Could not set input key "shading_diff" to group "Shading" in "<Pvwattsv8 object at 0x7fb225a574b0>".
Data is: [0] (<class 'list'>)
Received the following error: "Value must be numeric"

Expected Behavior

import ast
from pathlib import Path

import pandas as pd
from reV import Gen

site_df = pd.read_csv(HOME/'site_data.csv')
site_df['shading_azal'] = site_df['shading_azal'].apply(ast.literal_eval)

skyview_gen = Gen(technology='pvwattsv8',
                    project_points=pp_fpath.as_posix(),
                    sam_files=sam_fpath.as_posix(),
                    resource_file="/datasets/NSRDB/conus/nsrdb_conus_*_2022.h5",
                    output_request=output_reqs,
                    site_data=site_df,
                    gid_map=gid_map_fpath.as_posix(),
                    sites_per_worker=100,
                    memory_utilization_limit=0.4)

def main():
    skyview_gen = Gen(technology='pvwattsv8',
                      project_points=pp_fpath.as_posix(),
                      sam_files=sam_fpath.as_posix(),
                      resource_file="/datasets/NSRDB/conus/nsrdb_conus_*_2022.h5",
                      output_request=output_reqs,
                      site_data=site_df,
                      gid_map=gid_map_fpath.as_posix(),
                      sites_per_worker=100,
                      memory_utilization_limit=0.4)
    skyview_gen.run(out_fpath=out_fpath.as_posix())

Simplified outcome is to run generation with customized Python-native site data

System and installed versions

  • OS: Linux
  • Compute hardware: Kestrel
  • reV Version: 0.14.0
  • Python Version: 3.13.3
  • Pandas Version: 2.2.3

Charge code

There is no urgency, but will be a nice fix for site data inputs

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugfixFixes a bugp-highPriority: high

    Type

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions