Skip to content

Conversation

@kiranandcode
Copy link
Contributor

This PR adds documentation for the fact that weave.op is compatible with effectful and litellm.completions. In particular to get weave logging for template calls it simply suffices to add the @weave.op decorator to a template definition.

@weave.op
@Template.define
def haiku(topic: str) -> str:
  """Generate a haiku about {topic}."""
   raise NotHandled

Closes #493

@kiranandcode
Copy link
Contributor Author

The issue with this PR is that, as @weave.op wraps the template definition it no longer gets picked up as a tool for the LLM to use.

@weave.op
@Template.define
def haiku(topic: str) -> str:
    """Generate a haiku about the topic {topic}."""
    raise NotImplementedError

@weave.op
@Template.define
def poetry_compilation(topic: str) -> list[str]:
    """Generate a collection of poems surrounding {topic}, calling `haiku` on related topics to generate the elements."""
    raise NotImplementedError


with handler(LiteLLMProvider()):
    for poem in poetry_compilation('apple'):
        print(poem)
        print('')

@kiranandcode
Copy link
Contributor Author

Following discussions from #509, updating documentation to instead describe adding a custom handler.

@kiranandcode kiranandcode force-pushed the kg-llm-observability-documentation branch from a564ca3 to e37b992 Compare January 26, 2026 20:16
Copy link
Contributor

@eb8680 eb8680 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Closing #493 would require deleting the current logging handler from providers.py. Also, I'm just not sure we want to add weave as a library-level dependency in effectful[llm] or include this particular code snippet in the documentation when it's not something we've tested extensively ourselves - I can believe this code runs, but it's not associated with any unit tests and there's no way for someone to tell from this notebook cell if it solves the right problem and doesn't interact poorly with other components in surprising ways. I really think we should just leave out any recommendation like this unless it's something we've tested extensively ourselves or it's something built into LiteLLM.

@kiranandcode
Copy link
Contributor Author

okay sounds good. I'll drop the documentation change and drop the logging handler.

@kiranandcode kiranandcode force-pushed the kg-llm-observability-documentation branch from e37b992 to 77a0109 Compare January 27, 2026 15:23
@kiranandcode kiranandcode force-pushed the kg-llm-observability-documentation branch from 77a0109 to e532eed Compare January 27, 2026 15:27
@kiranandcode
Copy link
Contributor Author

@eb8680 dropped llmlogginghandler. Yixiu makes a lot of use of wandb, so I'll ask him to try out the wandb handler just in MARA for now and if it's popular and gets used internally sufficiently then we can document it later.

@eb8680 eb8680 changed the title Adds documentation regarding observability Remove LoggingLLMHandler Jan 27, 2026
@eb8680 eb8680 merged commit 783b8ba into master Jan 27, 2026
6 checks passed
@eb8680 eb8680 deleted the kg-llm-observability-documentation branch January 27, 2026 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consider offloading observability to LiteLLM

3 participants