reV version checks
Bug Description
When the reV submits site data to reVSAMWrapper, reV parses the site data into a dictionary and submits them. However, SAM expects certain forms of site data to be in Python native formats. For instance, in pvwattv8 module, if the user wants to pass the site data for shading group. Let us consider shading_azal, SAM expects a type sequence[sequence] - nested list. However, when the site data is saved as a CSV, nested lists are converted to object and even when reV loads them, it will be submitted to SAM as an object type. In the site data, the values are to be parsed before submitting to the SAM. Also, when the site data is loaded as a dataframe and parsed as a list, reV also raises an error. The full traceback is shared. I could find a conclusive evidence or specific line numbers where reV breaks with nested tuple.
Reproducible Example
reV generation -c config_gen.json
site_data.csv
sam.json
Full Traceback
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/SAM.py", line 418, in __setitem__
setattr(getattr(self.pysam, group), key, value)
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: Error (-4) converting nested tuple 0 into row in matrix.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/projects/rev/environments/revruns/lib/python3.13/concurrent/futures/process.py", line 254, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/generation.py", line 746, in _run_single_worker
raise e
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/generation.py", line 732, in _run_single_worker
out = cls.OPTIONS[tech].reV_run(
points_control,
...<6 lines>...
bias_correct=bias_correct,
)
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/generation.py", line 580, in reV_run
sim.run_gen_and_econ()
~~~~~~~~~~~~~~~~~~~~^^
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/generation.py", line 425, in run_gen_and_econ
self.run()
~~~~~~~~^^
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/generation.py", line 449, in run
self.assign_inputs()
~~~~~~~~~~~~~~~~~~^^
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/SAM.py", line 933, in assign_inputs
super().assign_inputs(copy.deepcopy(self.sam_sys_inputs))
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/SAM.py", line 610, in assign_inputs
self[k] = v
~~~~^^^
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/SAM.py", line 429, in __setitem__
raise SAMInputError(msg) from e
reV.utilities.exceptions.SAMInputError: Could not set input key "shading_azal" to group "Shading" in "<Pvwattsv8 object at 0x7f67633d0ed0>".
Data is: site_data (<class 'str'>)
Received the following error: "Error (-4) converting nested tuple 0 into row in matrix."
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<string>", line 1, in <module>
from gaps.cli.config import run_with_status_updates; from reV.generation.generation import Gen; su_args = "/kfs2/projects/pxs/erosenli/conus/rev", "generation", "rev_generation"; run_with_status_updates( Gen, {"analysis_years": [2022], "technology": "pvwattsv8", "output_request": ["cf_profile", "cf_profile_ac", "cf_mean", "cf_mean_ac", "ghi_mean", "lcoe_fcr", "ac", "dc", "clipped_power", "capital_cost", "fixed_operating_cost", "system_capacity", "system_capacity_ac", "fixed_charge_rate", "variable_operating_cost", "dc_ac_ratio"], "project_points": "/kfs2/projects/pxs/erosenli/conus/rev/project_points_inclusion_90m_13454.csv", "sam_files": {"default": "/kfs2/projects/pxs/erosenli/conus/rev/sam.json"}, "resource_file": "/datasets/NSRDB/conus/nsrdb_conus_*_2022.h5", "site_data": "/kfs2/projects/pxs/erosenli/conus/rev/site_data.csv", "gid_map": "/kfs2/projects/pxs/erosenli/conus/rev/gid_map.csv", "low_res_resource_file": None, "project_points_split_range": [0, 13454], "tag": "", "command_name": "generation", "pipeline_step": "generation", "config_file": "/kfs2/projects/pxs/erosenli/conus/rev/config_gen.json", "project_dir": "/kfs2/projects/pxs/erosenli/conus/rev", "job_name": "rev_generation", "out_dir": "/kfs2/projects/pxs/erosenli/conus/rev", "out_fpath": "/kfs2/projects/pxs/erosenli/conus/rev/rev_generation", "run_method": "run", "max_workers": None, "sites_per_worker": 100, "memory_utilization_limit": 0.8}, {"name": "rev_generation", "log_directory": "/kfs2/projects/pxs/erosenli/conus/rev/logs", "verbose": False, "node": True}, su_args, ["project_points"])
File "/projects/rev/environments/revruns/lib/python3.13/site-packages/gaps/cli/config.py", line 602, in run_with_status_updates
out = func(**run_kwargs)
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/generation.py", line 1146, in run
raise e
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/generation.py", line 1137, in run
self._parallel_run(
max_workers=max_workers,
...<2 lines>...
**kwargs,
)
^
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/base.py", line 1361, in _parallel_run
result = future.result(timeout=timeout)
File "/projects/rev/environments/revruns/lib/python3.13/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
~~~~~~~~~~~~~~~~~^^
File "/projects/rev/environments/revruns/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
reV.utilities.exceptions.SAMInputError: Could not set input key "shading_azal" to group "Shading" in "<Pvwattsv8 object at 0x7f67633d0ed0>".
Data is: site_data (<class 'str'>)
Received the following error: "Error (-4) converting nested tuple 0 into row in matrix.
WHEN SITE DATA IS PASSED AS NESTED LIST IN A DATAFRAME, IT STILL RAISES ERROR:
Traceback (most recent call last):
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/SAM.py", line 418, in __setitem__
setattr(getattr(self.pysam, group), key, value)
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: Value must be numeric
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/kfs2/projects/pxs/erosenli/conus/rev/gen.py", line 60, in <module>
main()
~~~~^^
File "/kfs2/projects/pxs/erosenli/conus/rev/gen.py", line 56, in main
skyview_gen.run(out_fpath=out_fpath.as_posix())
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/generation.py", line 1146, in run
raise e
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/generation.py", line 1123, in run
self.out = self._run_single_worker(pc_sub, **kwargs)
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/generation.py", line 746, in _run_single_worker
raise e
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/generation/generation.py", line 732, in _run_single_worker
out = cls.OPTIONS[tech].reV_run(
points_control,
...<6 lines>...
bias_correct=bias_correct,
)
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/generation.py", line 580, in reV_run
sim.run_gen_and_econ()
~~~~~~~~~~~~~~~~~~~~^^
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/generation.py", line 425, in run_gen_and_econ
self.run()
~~~~~~~~^^
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/generation.py", line 449, in run
self.assign_inputs()
~~~~~~~~~~~~~~~~~~^^
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/SAM.py", line 933, in assign_inputs
super().assign_inputs(copy.deepcopy(self.sam_sys_inputs))
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/SAM.py", line 610, in assign_inputs
self[k] = v
~~~~^^^
File "/kfs2/projects/rev/environments/gitrepos/reV/reV/SAM/SAM.py", line 429, in __setitem__
raise SAMInputError(msg) from e
reV.utilities.exceptions.SAMInputError: Could not set input key "shading_diff" to group "Shading" in "<Pvwattsv8 object at 0x7fb225a574b0>".
Data is: [0] (<class 'list'>)
Received the following error: "Value must be numeric"
Expected Behavior
import ast
from pathlib import Path
import pandas as pd
from reV import Gen
site_df = pd.read_csv(HOME/'site_data.csv')
site_df['shading_azal'] = site_df['shading_azal'].apply(ast.literal_eval)
skyview_gen = Gen(technology='pvwattsv8',
project_points=pp_fpath.as_posix(),
sam_files=sam_fpath.as_posix(),
resource_file="/datasets/NSRDB/conus/nsrdb_conus_*_2022.h5",
output_request=output_reqs,
site_data=site_df,
gid_map=gid_map_fpath.as_posix(),
sites_per_worker=100,
memory_utilization_limit=0.4)
def main():
skyview_gen = Gen(technology='pvwattsv8',
project_points=pp_fpath.as_posix(),
sam_files=sam_fpath.as_posix(),
resource_file="/datasets/NSRDB/conus/nsrdb_conus_*_2022.h5",
output_request=output_reqs,
site_data=site_df,
gid_map=gid_map_fpath.as_posix(),
sites_per_worker=100,
memory_utilization_limit=0.4)
skyview_gen.run(out_fpath=out_fpath.as_posix())
Simplified outcome is to run generation with customized Python-native site data
System and installed versions
- OS: Linux
- Compute hardware: Kestrel
- reV Version: 0.14.0
- Python Version: 3.13.3
- Pandas Version: 2.2.3
Charge code
There is no urgency, but will be a nice fix for site data inputs
reV version checks
I have confirmed this bug has not already been reported.
I have confirmed this bug exists on the latest version of reV.
I have confirmed this bug exists on the main branch of reV.
Bug Description
When the reV submits site data to reVSAMWrapper, reV parses the site data into a dictionary and submits them. However, SAM expects certain forms of site data to be in Python native formats. For instance, in pvwattv8 module, if the user wants to pass the site data for shading group. Let us consider shading_azal, SAM expects a type sequence[sequence] - nested list. However, when the site data is saved as a CSV, nested lists are converted to object and even when reV loads them, it will be submitted to SAM as an object type. In the site data, the values are to be parsed before submitting to the SAM. Also, when the site data is loaded as a dataframe and parsed as a list, reV also raises an error. The full traceback is shared. I could find a conclusive evidence or specific line numbers where reV breaks with nested tuple.
Reproducible Example
reV generation -c config_gen.json
site_data.csv
sam.json
Full Traceback
WHEN SITE DATA IS PASSED AS NESTED LIST IN A DATAFRAME, IT STILL RAISES ERROR:
Expected Behavior
Simplified outcome is to run generation with customized Python-native site data
System and installed versions
Charge code
There is no urgency, but will be a nice fix for site data inputs