fix: remove unsafe eval() in run_eval.py...

[orbisai0security]

Summary

Address high severity security finding in v1/experiments/long_horizon_benchmarks/run_eval.py.

Vulnerability

Field	Value
ID	python.lang.security.audit.eval-detected.eval-detected
Severity	HIGH
Scanner	semgrep
Rule	python.lang.security.audit.eval-detected.eval-detected
File	v1/experiments/long_horizon_benchmarks/run_eval.py:238
Assessment	Likely exploitable

Description: Detected the use of eval(). eval() can be dangerous if used to evaluate dynamic content. If this content can be input from outside the program, this may be a code injection vulnerability. Ensure evaluated content is not definable by external sources.

Evidence

Scanner confirmation: semgrep rule python.lang.security.audit.eval-detected.eval-detected flagged this pattern.

Production code: This file is in the production codebase, not test-only code.

Threat Model Context

This is a CLI tool - exploitation requires the attacker to control arguments or input files passed to the tool.

Changes

• v1/experiments/long_horizon_benchmarks/run_eval.py

Verification

• Build passes
• Scanner re-scan confirms fix
• LLM code review passed

Security Invariant

Property: The security boundary is maintained under adversarial input

Regression test

import pytest
import subprocess
import sys
import os


@pytest.mark.parametrize("payload", [
    "__import__('os').system('echo PWNED')",
    "__import__('subprocess').call(['id'])",
    "open('/etc/passwd').read()",
])
def test_eval_does_not_execute_arbitrary_code(payload):
    """Invariant: eval() usage in run_eval.py must not allow arbitrary code execution from external input."""
    # The run_eval.py file uses eval() which could be dangerous if fed external input.
    # We verify that passing adversarial strings as arguments does not result in code execution.
    script_path = os.path.join(
        os.path.dirname(__file__),
        "v1", "experiments", "long_horizon_benchmarks", "run_eval.py"
    )
    
    # If the script doesn't exist at relative path, try from repo root
    if not os.path.exists(script_path):
        script_path = os.path.join(
            "v1", "experiments", "long_horizon_benchmarks", "run_eval.py"
        )
    
    # Run the script with adversarial payload as an argument
    # The script should either reject the input, fail gracefully, or not execute the payload
    result = subprocess.run(
        [sys.executable, script_path, "--exp_name", payload],
        capture_output=True,
        text=True,
        timeout=10,
        env={**os.environ, "PYTHONDONTWRITEBYTECODE": "1"}
    )
    
    # The payload should never successfully execute - "PWNED" should not appear in output
    assert "PWNED" not in result.stdout, "Arbitrary code was executed via eval()"
    assert "PWNED" not in result.stderr, "Arbitrary code was executed via eval()"
    # Ensure no evidence of /etc/passwd content leaking
    assert "root:" not in result.stdout, "File read was executed via eval()"
    assert "root:" not in result.stderr, "File read was executed via eval()"

This test guards against regressions — it's useful independent of the code change above.

---

This change addresses a pattern flagged by static analysis. The code path handles user-influenced input and the fix reduces the attack surface against both manual and automated exploitation.

---

Automated security fix by OrbisAI Security