APPENG-3853 Prompt Standardization and Thinking mode changes #142

etsien · 2025-11-06T03:25:14Z

Builds on top of prior PR APPENG-3801-B

Consolidate all prompts into src/vuln_analysis/utils/prompting.py
Order them by stage section
Add system prompt sections for all stages, /no_think added by default for stages not requiring thinking mode
Added prompt helper functions to assemble into LangChain PromptTemplate or ChatPromptTemplate objects, based on if it's a simple stage or a ReAct Agent stage.
Cleaned up examples
Standardized prompt formatting, tool definitions, updated tool selection strategy to include latest transitive search updates, standardized output definitions.
Transitioned all stages' output parsing to use Pydantic2 for structured outputs types.
Added custom "thinking-aware" output parsing for ReAct stages to make more robust to output issues
Add the beginning of more detailed logging for error/failure tracking

… better clarity and quality

…l descriptions to both checklist and agent prompts

…o pick the class

adding in constants during the vdb generation check

…o pick the class

adding in constants during the vdb generation check

…cution-stages' of https://github.com/etsien/vulnerability-analysis into APPENG-3801-B-Agent-performance-fixes-checklist-and-execution-stages

output structure changes to come

…ging to stages too

etsien · 2025-11-06T04:03:37Z

/retest vulnerability-analysis-on-pr

tmihalac · 2025-11-10T13:10:25Z

Hi, I am running locally with your changes, I am getting this error:
KeyError: 'Input to ChatPromptTemplate is missing variables {\'\\n "scores"\'}. Expected: [\'\\n "scores"\', \'cve_data\'] Received: [\'cve_data\']\nNote: if you intended {\n "scores"} to be part of the string and not a variable, please escape it with double curly braces like: \'{{\n "scores"}}\'.\nFor troubleshooting, visit: https://python.langchain.com/docs/troubleshooting/errors/INVALID_PROMPT_INPUT ' During task with name 'calculate_intel_score_node' and id '92b3e313-3d0a-5350-bc2b-e4bd2a1bcf04'

When running on CVE-2024-26147
I am using config-http-openai.yml

When I disable calculate cve score feature, I am getting a different error:
ERROR: Exception in ASGI application

Exception Group Traceback (most recent call last):
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
| yield
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 177, in call
| async with anyio.create_task_group() as task_group:
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in aexit
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call
| await super().call(scope, receive, send)
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/applications.py", line 112, in call
| await self.middleware_stack(scope, receive, send)
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in call
| raise exc
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in call
| await self.app(scope, receive, _send)
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 176, in call
| with recv_stream, send_stream, collapse_excgroups():
| ^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 158, in exit
| self.gen.throw(value)
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
| raise exc
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 178, in call
| response = await self.dispatch_func(request, call_next)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/front_ends/fastapi/fastapi_front_end_plugin_worker.py", line 133, in authentication_log_filter
| return await self._suppress_authentication_logs(request, call_next)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/front_ends/fastapi/fastapi_front_end_plugin_worker.py", line 183, in _suppress_authentication_logs
| response = await call_next(request)
| ^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 156, in call_next
| raise app_exc
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 141, in coro
| await self.app(scope, receive_or_disconnect, send_no_error)
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in call
| await self.app(scope, receive, send)
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in call
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/routing.py", line 714, in call
| await self.middleware_stack(scope, receive, send)
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
| await route.handle(scope, receive, send)
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
| await self.app(scope, receive, send)
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
| response = await f(request)
| ^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
| raw_response = await run_endpoint_function(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
| return await dependant.call(**values)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/front_ends/fastapi/fastapi_front_end_plugin_worker.py", line 624, in post_single
| return await generate_single_response(payload, session_manager, result_type=result_type)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/front_ends/fastapi/response_helpers.py", line 117, in generate_single_response
| return await runner.result(to_type=result_type)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/runtime/runner.py", line 141, in result
| result = await self._entry_fn.ainvoke(self._input_message, to_type=to_type)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/builder/function.py", line 159, in ainvoke
| raise e
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/builder/function.py", line 149, in ainvoke
| result = await self._ainvoke(converted_input)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/builder/function.py", line 325, in _ainvoke
| return await self._ainvoke_fn(value)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/src/vuln_analysis/register.py", line 257, in _response_fn
| results = await graph.ainvoke(input_message)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/pregel/init.py", line 2389, in ainvoke
| async for chunk in self.astream(
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/pregel/init.py", line 2274, in astream
| async for _ in runner.atick(
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/pregel/runner.py", line 444, in atick
| await arun_with_retry(
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/pregel/retry.py", line 128, in arun_with_retry
| return await task.proc.ainvoke(task.input, config)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 583, in ainvoke
| input = await step.ainvoke(input, config, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 371, in ainvoke
| ret = await asyncio.create_task(coro, context=context)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/futures.py", line 289, in await
| yield self # This tells Task to wait for completion.
| ^^^^^^^^^^
| File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 385, in __wakeup
| future.result()
| File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/futures.py", line 202, in result
| raise self._exception.with_traceback(self._exception_tb)
| File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 314, in __step_run_and_handle_result
| result = coro.send(None)
| ^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/src/vuln_analysis/utils/error_handling_decorator.py", line 18, in catch_errors
| return await func(self, args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/src/vuln_analysis/register.py", line 198, in call_llm_engine_subgraph_node
| results = await subgraph.ainvoke(subgraph_input)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/pregel/init.py", line 2389, in ainvoke
| async for chunk in self.astream(
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/pregel/init.py", line 2274, in astream
| async for _ in runner.atick(
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/pregel/runner.py", line 444, in atick
| await arun_with_retry(
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/pregel/retry.py", line 128, in arun_with_retry
| return await task.proc.ainvoke(task.input, config)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 583, in ainvoke
| input = await step.ainvoke(input, config, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 371, in ainvoke
| ret = await asyncio.create_task(coro, context=context)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/futures.py", line 289, in await
| yield self # This tells Task to wait for completion.
| ^^^^^^^^^^
| File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 385, in __wakeup
| future.result()
| File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/futures.py", line 202, in result
| raise self._exception.with_traceback(self._exception_tb)
| File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 316, in __step_run_and_handle_result
| result = coro.throw(exc)
| ^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/src/vuln_analysis/register.py", line 141, in checklist_node
| return await cve_checklist_fn.ainvoke(state.model_dump())
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/builder/function.py", line 159, in ainvoke
| raise e
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/builder/function.py", line 149, in ainvoke
| result = await self._ainvoke(converted_input)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/builder/function.py", line 325, in _ainvoke
| return await self._ainvoke_fn(value)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/src/vuln_analysis/functions/cve_checklist.py", line 79, in _arun
| results = await asyncio.gather((generate_checklist_for_cve(cve_intel) for cve_intel in workflow_cve_intel))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 385, in __wakeup
| future.result()
| File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 314, in __step_run_and_handle_result
| result = coro.send(None)
| ^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/src/vuln_analysis/functions/cve_checklist.py", line 64, in generate_checklist_for_cve
| checklist = await generate_checklist(prompt=config.prompt,
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/src/vuln_analysis/utils/checklist_prompt_generator.py", line 170, in generate_checklist
| chat_prompt = ChatPromptTemplate.from_messages([
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langchain_core/prompts/chat.py", line 1170, in from_messages
| return cls(messages, template_format=template_format)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langchain_core/prompts/chat.py", line 952, in init
| _convert_to_message_template(message, template_format)
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langchain_core/prompts/chat.py", line 1447, in convert_to_message_template
| message = _create_template_from_message_type(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langchain_core/prompts/chat.py", line 1346, in _create_template_from_message_type
| message: BaseMessagePromptTemplate = HumanMessagePromptTemplate.from_template(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langchain_core/prompts/chat.py", line 437, in from_template
| prompt: Union[StringPromptTemplate, list] = PromptTemplate.from_template(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langchain_core/prompts/prompt.py", line 290, in from_template
| input_variables = get_template_variables(template, template_format)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langchain_core/prompts/string.py", line 255, in get_template_variables
| v for _, v, _, _ in Formatter().parse(template) if v is not None
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^
| ValueError: unmatched '{' in format spec
| During task with name 'checklist' and id '2f786271-fd7b-f583-427e-dc30e993125f'
| During task with name 'llm_engine' and id '3a61e2d2-cd4d-7ce3-1479-ae906e1939a8'

…d-thinking-mode-changes

etsien · 2025-11-11T14:51:43Z

/retest vulnerability-analysis-on-pr

…d-thinking-mode-changes

etsien · 2025-11-18T16:06:41Z

Hi, I am running locally with your changes, I am getting this error: KeyError: 'Input to ChatPromptTemplate is missing variables {\'\\n "scores"\'}. Expected: [\'\\n "scores"\', \'cve_data\'] Received: [\'cve_data\']\nNote: if you intended {\n "scores"} to be part of the string and not a variable, please escape it with double curly braces like: \'{{\n "scores"}}\'.\nFor troubleshooting, visit: https://python.langchain.com/docs/troubleshooting/errors/INVALID_PROMPT_INPUT ' During task with name 'calculate_intel_score_node' and id '92b3e313-3d0a-5350-bc2b-e4bd2a1bcf04'

When running on CVE-2024-26147 I am using config-http-openai.yml

When I disable calculate cve score feature, I am getting a different error: ERROR: Exception in ASGI application

* Exception Group Traceback (most recent call last):
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
  |     yield
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 177, in **call**
  |     async with anyio.create_task_group() as task_group:
  |                ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in **aexit**
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
  | Traceback (most recent call last):
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
  |     result = await app(  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in **call**
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in **call**
  |     await super().**call**(scope, receive, send)
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/applications.py", line 112, in **call**
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in **call**
  |     raise exc
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in **call**
  |     await self.app(scope, receive, _send)
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 176, in **call**
  |     with recv_stream, send_stream, collapse_excgroups():
  |                                    ^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 158, in **exit**
  |     self.gen.throw(value)
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
  |     raise exc
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 178, in **call**
  |     response = await self.dispatch_func(request, call_next)
  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/front_ends/fastapi/fastapi_front_end_plugin_worker.py", line 133, in authentication_log_filter
  |     return await self._suppress_authentication_logs(request, call_next)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/front_ends/fastapi/fastapi_front_end_plugin_worker.py", line 183, in _suppress_authentication_logs
  |     response = await call_next(request)
  |                ^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 156, in call_next
  |     raise app_exc
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 141, in coro
  |     await self.app(scope, receive_or_disconnect, send_no_error)
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in **call**
  |     await self.app(scope, receive, send)
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in **call**
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/routing.py", line 714, in **call**
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
  |     await route.handle(scope, receive, send)
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
  |     await self.app(scope, receive, send)
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
  |     response = await f(request)
  |                ^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
  |     raw_response = await run_endpoint_function(
  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
  |     return await dependant.call(**values)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/front_ends/fastapi/fastapi_front_end_plugin_worker.py", line 624, in post_single
  |     return await generate_single_response(payload, session_manager, result_type=result_type)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/front_ends/fastapi/response_helpers.py", line 117, in generate_single_response
  |     return await runner.result(to_type=result_type)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/runtime/runner.py", line 141, in result
  |     result = await self._entry_fn.ainvoke(self._input_message, to_type=to_type)
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/builder/function.py", line 159, in ainvoke
  |     raise e
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/builder/function.py", line 149, in ainvoke
  |     result = await self._ainvoke(converted_input)
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/builder/function.py", line 325, in _ainvoke
  |     return await self._ainvoke_fn(value)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/src/vuln_analysis/register.py", line 257, in _response_fn
  |     results = await graph.ainvoke(input_message)
  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/pregel/**init**.py", line 2389, in ainvoke
  |     async for chunk in self.astream(
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/pregel/**init**.py", line 2274, in astream
  |     async for _ in runner.atick(
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/pregel/runner.py", line 444, in atick
  |     await arun_with_retry(
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/pregel/retry.py", line 128, in arun_with_retry
  |     return await task.proc.ainvoke(task.input, config)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 583, in ainvoke
  |     input = await step.ainvoke(input, config, **kwargs)
  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 371, in ainvoke
  |     ret = await asyncio.create_task(coro, context=context)
  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/futures.py", line 289, in **await**
  |     yield self  # This tells Task to wait for completion.
  |     ^^^^^^^^^^
  |   File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 385, in __wakeup
  |     future.result()
  |   File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/futures.py", line 202, in result
  |     raise self._exception.with_traceback(self._exception_tb)
  |   File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 314, in __step_run_and_handle_result
  |     result = coro.send(None)
  |              ^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/src/vuln_analysis/utils/error_handling_decorator.py", line 18, in catch_errors
  |     return await func(self, _args, **kwargs)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/src/vuln_analysis/register.py", line 198, in call_llm_engine_subgraph_node
  |     results = await subgraph.ainvoke(subgraph_input)
  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/pregel/**init**.py", line 2389, in ainvoke
  |     async for chunk in self.astream(
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/pregel/**init**.py", line 2274, in astream
  |     async for _ in runner.atick(
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/pregel/runner.py", line 444, in atick
  |     await arun_with_retry(
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/pregel/retry.py", line 128, in arun_with_retry
  |     return await task.proc.ainvoke(task.input, config)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 583, in ainvoke
  |     input = await step.ainvoke(input, config, **kwargs)
  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 371, in ainvoke
  |     ret = await asyncio.create_task(coro, context=context)
  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/futures.py", line 289, in **await**
  |     yield self  # This tells Task to wait for completion.
  |     ^^^^^^^^^^
  |   File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 385, in __wakeup
  |     future.result()
  |   File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/futures.py", line 202, in result
  |     raise self._exception.with_traceback(self._exception_tb)
  |   File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 316, in __step_run_and_handle_result
  |     result = coro.throw(exc)
  |              ^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/src/vuln_analysis/register.py", line 141, in checklist_node
  |     return await cve_checklist_fn.ainvoke(state.model_dump())
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/builder/function.py", line 159, in ainvoke
  |     raise e
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/builder/function.py", line 149, in ainvoke
  |     result = await self._ainvoke(converted_input)
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/nat/builder/function.py", line 325, in _ainvoke
  |     return await self._ainvoke_fn(value)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/src/vuln_analysis/functions/cve_checklist.py", line 79, in _arun
  |     results = await asyncio.gather(_(generate_checklist_for_cve(cve_intel) for cve_intel in workflow_cve_intel))
  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 385, in __wakeup
  |     future.result()
  |   File "/home/tmihalac/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 314, in __step_run_and_handle_result
  |     result = coro.send(None)
  |              ^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/src/vuln_analysis/functions/cve_checklist.py", line 64, in generate_checklist_for_cve
  |     checklist = await generate_checklist(prompt=config.prompt,
  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/src/vuln_analysis/utils/checklist_prompt_generator.py", line 170, in generate_checklist
  |     chat_prompt = ChatPromptTemplate.from_messages([
  |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langchain_core/prompts/chat.py", line 1170, in from_messages
  |     return cls(messages, template_format=template_format)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langchain_core/prompts/chat.py", line 952, in **init**
  |     _convert_to_message_template(message, template_format)
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langchain_core/prompts/chat.py", line 1447, in _convert_to_message_template
  |     message_ = _create_template_from_message_type(
  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langchain_core/prompts/chat.py", line 1346, in _create_template_from_message_type
  |     message: BaseMessagePromptTemplate = HumanMessagePromptTemplate.from_template(
  |                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langchain_core/prompts/chat.py", line 437, in from_template
  |     prompt: Union[StringPromptTemplate, list] = PromptTemplate.from_template(
  |                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langchain_core/prompts/prompt.py", line 290, in from_template
  |     input_variables = get_template_variables(template, template_format)
  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/tmihalac/IdeaProjects/vulnerability-analysis/.venv/lib/python3.12/site-packages/langchain_core/prompts/string.py", line 255, in get_template_variables
  |     v for _, v, _, _ in Formatter().parse(template) if v is not None
  |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | ValueError: unmatched '{' in format spec
  | During task with name 'checklist' and id '2f786271-fd7b-f583-427e-dc30e993125f'
  | During task with name 'llm_engine' and id '3a61e2d2-cd4d-7ce3-1479-ae906e1939a8'

This should be resolved now. The prompting didn't escape parts of the {} json, so it was being intepreted as a variable. The prompting has been updated to escape these characters, and the secondary issue in the checklist has been addressed as well, passing the value instead of the text through to the next stage.

tmihalac · 2025-11-19T08:56:33Z

There are still a lot of calls to the transitive search with functions decided by he LLM:

2025-11-19 10:46:54,481 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - morpheus.vuln_analysis.utils.transitive_code_searcher_tool - DEBUG - Now using transitive code search tool with input =>helm.sh/helm/v3,LoadIndexFile 2025-11-19 10:46:54,482 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - vuln_analysis.utils.standard_library_cache - INFO - StdLibCache lookup (helm.sh/helm/v3): hits=13 misses=44 (eco=go) 2025-11-19 10:46:55,837 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - morpheus.vuln_analysis.utils.transitive_code_searcher_tool - DEBUG - Retriever found path=False path size=1
2025-11-19 10:46:55,838 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - morpheus.vuln_analysis.utils.transitive_code_searcher_tool - DEBUG - Now using transitive code search tool with input =>helm.sh/helm/v3,NewPluginGetter 2025-11-19 10:46:55,839 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - vuln_analysis.utils.standard_library_cache - INFO - StdLibCache lookup (helm.sh/helm/v3): hits=13 misses=45 (eco=go) 2025-11-19 10:46:55,838 - nat.builder.intermediate_step_manager - DEBUG - Popped end step 9310033c-c860-4b66-8b93-8134474f895a, name Call Chain Analyzer, type IntermediateStepType.FUNCTION_END, parent 5c3e2f3e-b5fc-4fcc-abfc-145ab0d7d3cb, stack id 139811709478912 2025-11-19 10:46:55,838 - nat.builder.intermediate_step_manager - DEBUG - Pushed start step 8145dc71-0231-4c28-839e-d70748b1a833, name Call Chain Analyzer, type IntermediateStepType.FUNCTION_START, parent 19fe860a-d04e-4f1b-bcb5-300beaaca5c5, stack id 139811709908992 2025-11-19 10:46:55,838 - nat.observability.exporter.span_exporter - DEBUG - Added span to tracking (outstanding: 8, stack: 8, event_id: 8145dc71-0231-4c28-839e-d70748b1a833) 2025-11-19 10:46:56,264 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - morpheus.vuln_analysis.utils.transitive_code_searcher_tool - DEBUG - Retriever found path=True path size=5
2025-11-19 10:47:05,127 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - morpheus.vuln_analysis.utils.transitive_code_searcher_tool - DEBUG - Now using transitive code search tool with input =>gopkg.in/yaml.v2,yaml.Unmarshal 2025-11-19 10:47:05,127 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - vuln_analysis.utils.standard_library_cache - INFO - StdLibCache lookup (gopkg.in/yaml.v2): hits=13 misses=46 (eco=go) 2025-11-19 10:47:05,137 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - morpheus.vuln_analysis.utils.chain_of_calls_retriever - ERROR - Cannot find initial function=Unmarshal, in package=gopkg.in/yaml.v2 2025-11-19 10:47:05,137 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - morpheus.vuln_analysis.utils.transitive_code_searcher_tool - DEBUG - Retriever found path=False path size=0
2025-11-19 10:47:10,153 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - morpheus.vuln_analysis.utils.transitive_code_searcher_tool - DEBUG - Now using transitive code search tool with input =>github.com/helm/helm/v3/pkg/action,RollbackRelease 2025-11-19 10:47:10,153 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - vuln_analysis.utils.standard_library_cache - INFO - StdLibCache lookup (github.com/helm/helm/v3/pkg/action): hits=13 misses=47 (eco=go) 2025-11-19 10:47:10,209 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - morpheus.vuln_analysis.utils.transitive_code_searcher_tool - DEBUG - Retriever found path=False path size=1
2025-11-19 10:47:53,763 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - morpheus.vuln_analysis.utils.transitive_code_searcher_tool - DEBUG - Now using transitive code search tool with input =>gopkg.in/yaml.v2,yaml.Unmarshal 2025-11-19 10:47:53,763 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - vuln_analysis.utils.standard_library_cache - INFO - StdLibCache lookup (gopkg.in/yaml.v2): hits=14 misses=49 (eco=go) 2025-11-19 10:47:53,774 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - morpheus.vuln_analysis.utils.chain_of_calls_retriever - ERROR - Cannot find initial function=Unmarshal, in package=gopkg.in/yaml.v2 2025-11-19 10:47:53,775 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - morpheus.vuln_analysis.utils.transitive_code_searcher_tool - DEBUG - Retriever found path=False path size=0
2025-11-19 10:47:56,322 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - morpheus.vuln_analysis.utils.transitive_code_searcher_tool - DEBUG - Now using transitive code search tool with input =>helm.sh/helm/v3/pkg/repo,LoadIndexFile 2025-11-19 10:47:56,322 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - vuln_analysis.utils.standard_library_cache - INFO - StdLibCache lookup (helm.sh/helm/v3/pkg/repo): hits=14 misses=50 (eco=go) 2025-11-19 10:47:56,675 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - morpheus.vuln_analysis.utils.transitive_code_searcher_tool - DEBUG - Retriever found path=False path size=1

The analysis result is 'requires_environment':
2025-11-19 10:07:09,669 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - vuln_analysis.utils.error_handling_decorator - DEBUG - Before running tool container_image_analysis_data in file vuln_analysis.tools.container_image_analysis_data 2025-11-19 10:07:09,681 - threadName=MainThread- traceId=9949142a2dbdba9cfee8d2fd8cf15eb8 - vuln_analysis.utils.llm_engine_utils - INFO - Vulnerability 'CVE-2024-26147' affected status: FALSE. Label: requires_environment. CVSS score: Failed to generate a CVSS score from vector string. Error: Malformed CVSS3 field "Agent stopped due to iteration limit or time limit."

Here is the UI's text:

requires_environment
Reason
The Helm client is present, but there is no evidence of malicious plugins or untrusted input data (crafted yaml files) that could exploit the vulnerability, indicating the required environment for exploitation is absent.
Summary
The CVE is not exploitable in this container. Investigation revealed that while the Helm client is used within the container image, there is no conclusive evidence to suggest that malicious plugins are present that could cause the Helm client to panic. Additionally, the investigation did not find any evidence of the Helm SDK being used to process potentially untrusted input data that could include crafted yaml files designed to exploit this vulnerability. Although the investigation did not confirm the Helm version in use, the lack of malicious plugins and untrusted input data reduces the risk of exploitation.
CVSS Vector String
CVSS:3.1/AV:N/AC:H/PR:H/Agent stopped due to iteration limit or time limit./S:C/C:L/I:L/A:N
CVSS Score
Failed to generate a CVSS score from vector string. Error: Malformed CVSS3 field "Agent stopped due to iteration limit or time limit."
CVE intel score
91
Checklist:
Q: Is the Helm SDK used within the container image, specifically the `LoadIndexFile`, `DownloadIndexFile`, or `LoadDir` functions in the `repo` or `plugin` packages, which are vulnerable to the uninitialized variable issue when parsing index and plugin yaml files missing expected content? Check for imports or calls to these functions in the codebase, particularly in areas handling Helm operations or plugin management
A: Agent stopped due to iteration limit or time limit.
Q: Does the application process potentially untrusted input data that could include crafted yaml files designed to exploit this vulnerability, such as missing metadata in `index.yaml` or `plugin.yaml` files? Evaluate how the application handles exceptions or panics that may occur due to this issue
A: Agent stopped due to iteration limit or time limit.
Q: Are there any malicious plugins present in the container image that could be causing the Helm client to panic, and if so, can they be manually removed from the filesystem to mitigate the issue? Inspect the plugin directory and configuration to identify any potentially malicious plugins
A: There is no conclusive evidence to suggest that there are malicious plugins present in the container image that could be causing the Helm client to panic. However, the investigation reveals several vulnerabilities related to the Helm client and plugins, which could potentially be exploited by an attacker. It is recommended to update the Helm client to the latest version and to carefully review the plugins used in the container image to ensure they are not malicious. Additionally, it is recommended to implement security best practices, such as validating user input and using secure protocols for communication, to prevent potential attacks.
Q: Has the Helm version been updated to 3.14.2 or later, which resolves the uninitialized variable vulnerability? Check the Helm version in use within the container image and consider updating to a patched version if necessary
A: I do not have a definitive answer for this checklist item.
Q: Are there any error handling mechanisms in place, such as using `recover` to catch panics, when calling the affected functions in the Helm SDK? Review the code to ensure that proper error handling is implemented to prevent crashes or unexpected behavior
A: Agent stopped due to iteration limit or time limit.```

etsien added 30 commits October 2, 2025 16:13

bugfix in the testing env

a1834bd

update tool descriptions for clarity

1d79035

refactor tool names to be class constants instead of disparate strings

6864fa7

add initial unit tests

e05ea7a

rename tool names to be more consistent and distinct

8893f5c

update unit tests with tool names and tool constants

dd18463

cleanup startup guide notebook

4efffcd

rework intel source score section

8f3182e

update agent execution stage prompts and make tool descriptions dynamic

dd215cf

add tests for dynamic tool descriptions

35ee318

revamp the tool description list, as well as the checklist prompt for…

0af8e7a

… better clarity and quality

revamp checklist prompt implementation, as well as add in dynamic too…

a882f88

…l descriptions to both checklist and agent prompts

update tests for tool descriptions

26f0d74

add more detailed agent examples with more useful MRKL-formatted steps

186350d

update for summary prompt

f71e9f8

update justification prompt with more logic and explanations on how t…

9707671

…o pick the class

update CVSS prompts and cleanup examples and guidance

faeb811

bugfix on intel source

dcf836f

bug patch for vdb generation

efa84ad

adding in constants during the vdb generation check

bugfix by Tamar

be0b27d

update register_function() and transitive_search() descriptions

538257d

bugfix in the testing env

36bb6d3

update tool descriptions for clarity

7b7695d

refactor tool names to be class constants instead of disparate strings

d105a38

add initial unit tests

09fb531

rename tool names to be more consistent and distinct

f475fb2

update unit tests with tool names and tool constants

a4ed889

cleanup startup guide notebook

3a13650

rework intel source score section

3b477b0

update agent execution stage prompts and make tool descriptions dynamic

dcdb5cf

etsien added 19 commits November 5, 2025 11:59

add more detailed agent examples with more useful MRKL-formatted steps

a5368b6

update for summary prompt

dbab156

update justification prompt with more logic and explanations on how t…

f585abb

…o pick the class

update CVSS prompts and cleanup examples and guidance

b3b53e1

bugfix on intel source

c26936e

bug patch for vdb generation

ee0c6af

adding in constants during the vdb generation check

bugfix by Tamar

309a554

update register_function() and transitive_search() descriptions

40de82e

add function locator descriptions

7e0bddb

add names to configs

eb09d06

add local output for local testing

8754762

Merge branch 'APPENG-3801-B-Agent-performance-fixes-checklist-and-exe…

3449499

…cution-stages' of https://github.com/etsien/vulnerability-analysis into APPENG-3801-B-Agent-performance-fixes-checklist-and-execution-stages

move all prompts out of other files and into prompting.py

ee9fd3c

Update prompts, structure them, and order them based on pipeline stage

aded7f2

output structure changes to come

add pydantic fields for pipeline stages

2b282d0

add custom ReAct/thinking aware parser to robustly handle ReAct outputs

1503e07

add pydantic output parsing and structure to pipeline stages, add log…

508c829

…ging to stages too

add pydantic to pipeline checklist

d3c9ee5

bugfix

710743a

Update tool_names.py

5986ced

Merge branch 'rh-aiq-main' into APPENG-3853-prompt-standardization-an…

26f291a

…d-thinking-mode-changes

etsien requested a review from zvigrinberg November 10, 2025 19:29

zvigrinberg mentioned this pull request Nov 18, 2025

APPENG-3801-B - Agent performance fixes - all agent stages #134

Merged

etsien added 2 commits November 18, 2025 10:53

bugfix for brackets and variable passing issues

317ba90

Merge branch 'rh-aiq-main' into APPENG-3853-prompt-standardization-an…

b364dd3

…d-thinking-mode-changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

APPENG-3853 Prompt Standardization and Thinking mode changes #142

APPENG-3853 Prompt Standardization and Thinking mode changes #142

Uh oh!

etsien commented Nov 6, 2025 •

edited

Loading

Uh oh!

etsien commented Nov 6, 2025

Uh oh!

tmihalac commented Nov 10, 2025 •

edited

Loading

Uh oh!

etsien commented Nov 11, 2025

Uh oh!

etsien commented Nov 18, 2025

Uh oh!

tmihalac commented Nov 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

APPENG-3853 Prompt Standardization and Thinking mode changes #142

Are you sure you want to change the base?

APPENG-3853 Prompt Standardization and Thinking mode changes #142

Uh oh!

Conversation

etsien commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

etsien commented Nov 6, 2025

Uh oh!

tmihalac commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

etsien commented Nov 11, 2025

Uh oh!

etsien commented Nov 18, 2025

Uh oh!

tmihalac commented Nov 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

etsien commented Nov 6, 2025 •

edited

Loading

tmihalac commented Nov 10, 2025 •

edited

Loading