Skip to content

reconstruct tool calls #30

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 16, 2025
Merged

reconstruct tool calls #30

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions cachy.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -330,3 +330,8 @@
{"key": "5c8485a6", "response": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-sonnet-4-5-20250929\",\"id\":\"msg_01FAdRVTQeQTovybY7EVW2Fr\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"stop_reason\":null,\"stop_sequence\":null,\"usage\":{\"input_tokens\":4,\"cache_creation_input_tokens\":14,\"cache_read_input_tokens\":1622,\"cache_creation\":{\"ephemeral_5m_input_tokens\":14,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":9,\"service_tier\":\"standard\"}} }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"Goodbye! Feel free to come back if\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" you have any questions.\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\",\"stop_sequence\":null},\"usage\":{\"input_tokens\":4,\"cache_creation_input_tokens\":14,\"cache_read_input_tokens\":1622,\"output_tokens\":17}}\n\nevent: message_stop\ndata: {\"type\":\"message_stop\"}\n\n"}
{"key": "fc5ede02", "response": "{\"model\":\"claude-sonnet-4-20250514\",\"id\":\"msg_01M2HV41ZonZ1AAJg5FUy6zr\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[{\"type\":\"text\",\"text\":\"\\n\\nThe result of 47 + 23 is 70.\"}],\"stop_reason\":\"end_turn\",\"stop_sequence\":null,\"usage\":{\"input_tokens\":573,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":18,\"service_tier\":\"standard\"}}"}
{"key": "df4ef045", "response": "{\"model\":\"claude-sonnet-4-20250514\",\"id\":\"msg_0139iCPpDfH1itSAtoBGNV4V\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[{\"type\":\"text\",\"text\":\"\\n\\nThe answer is **129**.\\n\\nI calculated this by first adding 47 + 23 = 70, then adding 70 + 59 = 129.\"}],\"stop_reason\":\"end_turn\",\"stop_sequence\":null,\"usage\":{\"input_tokens\":702,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":41,\"service_tier\":\"standard\"}}"}
{"key": "d4589fc2", "response": "{\"model\":\"claude-sonnet-4-5-20250929\",\"id\":\"msg_019GSku9pTraoyhUZZxkDbws\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[{\"type\":\"tool_use\",\"id\":\"toolu_01MCSt43tUWZKqaoFXbn7krH\",\"name\":\"simple_add\",\"input\":{\"a\":5,\"b\":3}}],\"stop_reason\":\"tool_use\",\"stop_sequence\":null,\"usage\":{\"input_tokens\":618,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":70,\"service_tier\":\"standard\"}}"}
{"key": "20a4c712", "response": "{\"model\":\"claude-sonnet-4-5-20250929\",\"id\":\"msg_01ErxCc3aGgypa4ndgTwMuMf\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[{\"type\":\"text\",\"text\":\"\\n\\nThe result of 5 + 3 is **8**.\"}],\"stop_reason\":\"end_turn\",\"stop_sequence\":null,\"usage\":{\"input_tokens\":742,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":18,\"service_tier\":\"standard\"}}"}
{"key": "71b7f751", "response": "{\"model\":\"claude-sonnet-4-5-20250929\",\"id\":\"msg_01YPXAnNErrUzxUZpLxqP7rT\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[{\"type\":\"text\",\"text\":\"Here's a joke based on the number 8:\\n\\nWhy was 6 afraid of 7?\\n\\nBecause 7 8 (ate) 9!\\n\\nBut since we got 8 as our answer, here's another one:\\n\\nWhat did the number 8 say when it looked in the mirror?\\n\\n\\\"Nice belt!\\\"\\n\\n(Because 8 looks like it's wearing a belt around its middle! \ud83d\ude04)\"}],\"stop_reason\":\"end_turn\",\"stop_sequence\":null,\"usage\":{\"input_tokens\":774,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":97,\"service_tier\":\"standard\"}}"}
{"key": "8ff05b20", "response": "{\"model\":\"claude-sonnet-4-5-20250929\",\"id\":\"msg_01UkSZXczvtptdBLJemBkwnv\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[{\"type\":\"text\",\"text\":\"# Image Description\\n\\nThis adorable image shows a **Cavalier King Charles Spaniel puppy** with the classic Blenheim coloring (chestnut brown and white markings). \\n\\n## Key features visible:\\n- **Expressive brown eyes** looking directly at the camera\\n- **Soft, fluffy ears** with rich brown fur\\n- **White blaze** down the center of the face\\n- **White chest and paws**\\n- The puppy is lying on **green grass**\\n- **Purple flowers** (appear to be asters or similar) in the background\\n- Warm, soft lighting creating a charming portrait effect\\n\\nThe puppy has that irresistibly sweet, gentle expression that Cavalier King Charles Spaniels are famous for. This looks like a professional or carefully composed photograph, possibly for a breeder, pet portrait, or greeting card.\"}],\"stop_reason\":\"end_turn\",\"stop_sequence\":null,\"usage\":{\"input_tokens\":105,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":195,\"service_tier\":\"standard\"}}"}
{"key": "130a52f1", "response": "{\"model\":\"claude-sonnet-4-5-20250929\",\"id\":\"msg_01NsVrovfY7JrTr5dPygRJhb\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[{\"type\":\"text\",\"text\":\" D A C T E D\\n\\nI don't actually know your name - you haven't told me what it is yet! If you'd like me to spell your name, please let me know what it is first.\"}],\"stop_reason\":\"end_turn\",\"stop_sequence\":null,\"usage\":{\"input_tokens\":16,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":47,\"service_tier\":\"standard\"}}"}
Comment on lines +336 to +337
Copy link
Contributor Author

@comhar comhar Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although these 2 items are unrelated to the pr, they were automatically added to the cache because their chat history is linked to upstream calls. This dependency has been removed in this pr.

2 changes: 2 additions & 0 deletions lisette/_modidx.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,10 @@
'lisette.core.Chat.print_hist': ('core.html#chat.print_hist', 'lisette/core.py'),
'lisette.core._add_cache_control': ('core.html#_add_cache_control', 'lisette/core.py'),
'lisette.core._alite_call_func': ('core.html#_alite_call_func', 'lisette/core.py'),
'lisette.core._build_tool_hist': ('core.html#_build_tool_hist', 'lisette/core.py'),
'lisette.core._bytes2content': ('core.html#_bytes2content', 'lisette/core.py'),
'lisette.core._clean_str': ('core.html#_clean_str', 'lisette/core.py'),
'lisette.core._details_extract': ('core.html#_details_extract', 'lisette/core.py'),
'lisette.core._has_cache': ('core.html#_has_cache', 'lisette/core.py'),
'lisette.core._has_search': ('core.html#_has_search', 'lisette/core.py'),
'lisette.core._lite_call_func': ('core.html#_lite_call_func', 'lisette/core.py'),
Expand Down
75 changes: 49 additions & 26 deletions lisette/core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
# AUTOGENERATED! DO NOT EDIT! File to edit: ../nbs/00_core.ipynb.

# %% auto 0
__all__ = ['sonn45', 'effort', 'patch_litellm', 'mk_msg', 'mk_msgs', 'stream_with_complete', 'lite_mk_func', 'cite_footnote',
'cite_footnotes', 'Chat', 'random_tool_id', 'mk_tc', 'mk_tc_req', 'mk_tc_result', 'mk_tc_results',
__all__ = ['sonn45', 'effort', 'patch_litellm', 'mk_msg', 'stream_with_complete', 'lite_mk_func', 'cite_footnote',
'cite_footnotes', 'Chat', 'random_tool_id', 'mk_tc', 'mk_tc_req', 'mk_tc_result', 'mk_tc_results', 'mk_msgs',
'astream_with_complete', 'AsyncChat', 'aformat_stream', 'adisplay_stream']

# %% ../nbs/00_core.ipynb
Expand Down Expand Up @@ -117,23 +117,6 @@ def mk_msg(content, # Content: str, bytes (image), list of mixed content, o
msg = {"role": role, "content": c}
return _add_cache_control(msg, ttl=ttl) if cache else msg

# %% ../nbs/00_core.ipynb
def mk_msgs(msgs, # List of messages (each: str, bytes, list, or dict w 'role' and 'content' fields)
cache=False, # Enable Anthropic caching
ttl=None, # Cache TTL: '5m' (default) or '1h'
):
"Create a list of LiteLLM compatible messages."
if not msgs: return []
if not isinstance(msgs, list): msgs = [msgs]
res,role = [],'user'
for m in msgs:
res.append(msg:=mk_msg(m, role=role))
role = 'assistant' if msg['role'] in ('user','function', 'tool') else 'user'
if cache:
res[-1] = _add_cache_control(res[-1], ttl)
res[-2] = _add_cache_control(res[-2], ttl)
return res

# %% ../nbs/00_core.ipynb
def stream_with_complete(gen, postproc=noop):
"Extend streaming response chunks with the complete response"
Expand Down Expand Up @@ -269,11 +252,9 @@ def random_tool_id():
return f'toolu_{random_part}'

# %% ../nbs/00_core.ipynb
def mk_tc(func, idx=1, **kwargs):
args = json.dumps(kwargs)
if callable(func): func = func.__name__
id = random_tool_id()
return {'index': idx, 'function': {'arguments': args, 'name': func}, 'id': id, 'type': 'function'}
def mk_tc(func, args, tcid=None, idx=1):
if not tcid: tcid = random_tool_id()
return {'index': idx, 'function': {'arguments': args, 'name': func}, 'id': tcid, 'type': 'function'}
Comment on lines +255 to +257
Copy link
Contributor Author

@comhar comhar Oct 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes make it easy to create tool messages from the tool call summary message (i.e. when the called function and args are strings).

The downside is that there's a little more effort involved in creating a tool call message when the function and args are symbols.

For example. Here's the syntax on main

mk_tc(simple_add, a=5, b=7)

vs the syntax for this pr.

mk_tc(simple_add.__name__, json.dumps(dict(a=5, b=7)))


# %% ../nbs/00_core.ipynb
def mk_tc_req(content, tcs):
Expand All @@ -287,6 +268,48 @@ def mk_tc_result(tc, result): return {'tool_call_id': tc['id'], 'role': 'tool',
# %% ../nbs/00_core.ipynb
def mk_tc_results(tcq, results): return [mk_tc_result(a,b) for a,b in zip(tcq.tool_calls, results)]

# %% ../nbs/00_core.ipynb
def _details_extract(x):
"Extract fn, args, tool_call_id, result from <details>"
m = re.search(r'<details.*?>(.*?)</details>', x, re.DOTALL)
tc, tcid, res = re.findall(r'-\s*`([^`]+)`', m.group(1))
fn, args = re.search(r'(\w+)\((.*?)\)', tc).groups()
return fn, args, res, tcid
Comment on lines +272 to +277
Copy link
Contributor Author

@comhar comhar Oct 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pr modifies the tool call summary message by including a tool call id in the details section.

As a result, _details_extract will throw an error if it runs on a tool call summary message generated with the current version of lisette.

That version generates a message like this

I'll use the `addy` function to add 5 and 3 for you.
<details class='tool-usage-details'>

 `addy({"a": 5, "b": 3})`
  - `8`

</details>

The result is 8.

Whereas this pr expects this structure

I'll use the `addy` function to sum 5 and 7 for you.
<details class='tool-usage-details'>

 - `simple_add({"a": 5, "b": 7})`
 - `12`
 - `toolu_01RPbSeouj8mc2N4rfjw2BaH`

</details>

The sum of 5 and 7 is **12**.

We could make it backwards compatible by using a random tool call id? Maybe this is overkill?

Copy link
Contributor

@jph00 jph00 Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using regexen here doesn't feel robust to me. I'd have thought that using json would be better - i.e. a proper structured format with a well-tested standard implementation. wdyt?


# %% ../nbs/00_core.ipynb
def _build_tool_hist(msg):
"Build original tool call messages from the tool call summary."
hist = []
parts = re.split(r'(<details.*?>.*?</details>)', msg, flags=re.DOTALL)
for islast, (i,o) in loop_last(enumerate(parts)):
if "<details class='tool-usage-details'>" not in o:
if islast: hist.append(o.strip())
continue
fn, args, res, tcid = _details_extract(o)
tc = mk_tc(fn, args, tcid)
tcq = mk_tc_req(parts[i-1].strip() if i>0 else "", [tc])
tcr = first(mk_tc_results(tcq, [res]))
hist.extend([tcq, tcr])
return hist

# %% ../nbs/00_core.ipynb
def mk_msgs(msgs, # List of messages (each: str, bytes, list, or dict w 'role' and 'content' fields)
Copy link
Contributor Author

@comhar comhar Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redefining mk_msgs for the sake of a 1 line change isn't ideal. Is there a better way to do this?

Copy link
Contributor

@jph00 jph00 Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah the implementation feels a bit over-clever to me. It needn't be so integrated and automatic. It's quite special-case behaviour. Instead, I'd expect to have a function like "extract_tcs()" which you pass a message to, and it turns it into a list of messages with tool calls expanded.

Copy link
Contributor Author

@comhar comhar Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We went back and forth on this. We opted for the automatic implementation because we didn't see a strong reason not to expand these messages.

For extract_tcs() would that be applied before passing hist to Chat? Maybe we could add a param extract_tcs to Chat which would automatically expand messages using the extract_tcs fn?

cache=False, # Enable Anthropic caching
ttl=None, # Cache TTL: '5m' (default) or '1h'
):
"Create a list of LiteLLM compatible messages."
if not msgs: return []
if not isinstance(msgs, list): msgs = [msgs]
res,role = [],'user'
msgs = L(msgs).map(lambda m: _build_tool_hist(m) if "<details class='tool-usage-details'>" in m else [m]).concat()
Copy link
Contributor Author

@comhar comhar Oct 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line ensures that we automatically reconstruct the tool call history for every tool call summary msg in msgs.

for m in msgs:
res.append(msg:=mk_msg(m, role=role))
role = 'assistant' if msg['role'] in ('user','function', 'tool') else 'user'
if cache:
res[-1] = _add_cache_control(res[-1], ttl)
res[-2] = _add_cache_control(res[-2], ttl)
return res

# %% ../nbs/00_core.ipynb
async def _alite_call_func(tc, ns, raise_on_err=True):
try: fargs = json.loads(tc.function.arguments)
Expand Down Expand Up @@ -385,9 +408,9 @@ async def aformat_stream(rs, include_usage=False):
if include_usage: yield f"\nUsage: {o.usage}"
if (c := getattr(o.choices[0].message, 'tool_calls', None)):
fn = first(c).function
yield f"\n<details class='tool-usage-details'>\n\n `{fn.name}({_trunc_str(fn.arguments)})`\n"
yield f"\n<details class='tool-usage-details'>\n\n - `{fn.name}({_trunc_str(fn.arguments, replace='<TRUNCATED>')})`\n"
elif isinstance(o, dict) and 'tool_call_id' in o:
yield f" - `{_trunc_str(_clean_str(o.get('content')))}`\n\n</details>\n\n"
yield f" - `{o['tool_call_id']}`\n\n - `{_trunc_str(_clean_str(o.get('content')),replace='<TRUNCATED>')}`\n\n</details>\n\n"
Copy link
Contributor Author

@comhar comhar Oct 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We needed to include the tool_call_id in the summary so that we could fully reproduce the original tool call message. If we used a random id, it would bust the LLM cache and cachy's cache 😅 .


# %% ../nbs/00_core.ipynb
async def adisplay_stream(rs):
Expand Down
Loading