Skip to content

aws_bedrock: tolerate non-object elements in invocation output.outputBodyJson lists #15726

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+168 −1
Open

aws_bedrock: tolerate non-object elements in invocation output.outputBodyJson lists #15726

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/aws_bedrock/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
- version: "1.4.0"
changes:
- description: Tolerate non-object elements in invocation `output.outputBodyJson` lists.
type: enhancement
link: https://github.com/elastic/integrations/pull/15726
- version: "1.3.0"
changes:
- description: Improve documentation to align with new guidelines.
Expand Down
1 change: 1 addition & 0 deletions packages/aws_bedrock/data_stream/invocation/_dev/test/pipeline/test-aws-bedrock.log
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@
{"schemaType":"ModelInvocationLog","schemaVersion":"1.0","timestamp":"2024-10-11T12:15:04Z","accountId":"123456789012","identity":{"arn":"arn:aws:sts::123456789012:assumed-role/DUMMYROLE/dummyuser"},"region":"us-west-2","requestId":"abcdef12-3456-7890-abcd-ef1234567890","operation":"ConverseStream","modelId":"dummy-model-v1","input":{"inputContentType":"application/json","inputBodyJson":{"messages":[{"role":"user","content":[{"text":"This is a dummy input message"},{"text":"Here is another part of the dummy message"}]},{"role":"assistant","content":[{"text":"This is a dummy assistant response"},{"text":"Additional response content"}]},{"role":"user","content":[{"text":"Another dummy question"},{"text":"Follow-up to the dummy question"}]}],"inferenceConfig":{"maxTokens":300,"temperature":1,"topP":0.999,"stopSequences":["\n\nHuman:"]},"additionalModelRequestFields":{"top_k":250}},"inputTokenCount":320},"output":{"outputContentType":"application/json","outputBodyJson":{"output":{"message":{"role":"assistant","content":[{"text":"Here is a dummy response for testing purposes. Always ensure you're using valid data for your operations."},{"text":"Another part of the dummy response"}]}},"stopReason":"max_tokens","metrics":{"latencyMs":5561},"usage":{"inputTokens":320,"outputTokens":300,"totalTokens":620}},"outputTokenCount":300}}
{"schemaType":"ModelInvocationLog","schemaVersion":"1.0","timestamp":"2025-01-09T18:49:23Z","accountId":"795142007471","identity":{"arn":"arn:aws:sts::795142007471:assumed-role/BEDROCKPLAYGROUNDACCESS/uwga"},"region":"us-east-1","requestId":"19926a63-3c07-4481-864c-49516d208539","operation":"ConverseStream","modelId":"anthropic.claude-v2:1","input":{"inputContentType":"application/json","inputBodyJson":{"messages":[{"role":"user","content":[{"document":{"format":"pdf","name":"API Design General Guidelines_e5c8d7dab1cd44e9af902bb9515c2d49-090125-1321-342","source":{"s3Uri":"s3://vgi-ead-eng-us-east-1-bedrock-logging/AWSLogs/795142007471/BedrockModelInvocationLogs/us-east-1/2025/01/09/18/data/19926a63-3c07-4481-864c-49516d208539_input_media_00.pdf"}}},{"text":"Can you give me some key takeways to look for in a api spec yaml file?"}]},{"role":"assistant","content":[{"text":"Here are some key takeaways for reviewing an API spec YAML file:\n\n1. Title - Should clearly describe the API's purpose and resources. Avoid technical terms or abbreviations.\n\n2. Base Path - Should represent the main resource the API manages. Use plural nouns and hyphens.\n\n3. Endpoints - Use nouns instead of verbs, hyphens instead of spaces/underscores, and appropriate HTTP methods.\n\n4. Response Structure - Responses should wrap data arrays in an outer object for easier expanding, clarity, and parsing.\n\n5. Model Names - Use PascalCase for models/schemas and camelCase for fields. Be descriptive and avoid abbreviations.\n\n6. Enums - Define possible values in the spec. Exchange enum values rather than indexes in requests/responses.\n\n7. Allowed Value Fields - Define as strings and describe where to find possible values.\n\n8. Tags - Use to organize endpoints. Root tag should be product family. Endpoint tags can further categorize.\n\n9. Pagination - Use limit/offset parameters and structure requests/responses accordingly.\n\n10. IDs - Include IDs and timestamps for created/updated resources in responses."}]},{"role":"user","content":[{"text":"using this, can you give corrections or suggestions to a given api-spec.yaml file?"}]},{"role":"assistant","content":[{"text":"Unfortunately I cannot provide specific suggestions or corrections to an API spec file without seeing the actual file contents. However, I can make some general recommendations:\n\n- Share the api-spec.yaml file or relevant snippets you would like me to take a look at.\n\n- I would be happy to review portions of the spec and provide feedback based on the takeaways I mentioned earlier. For example, checking if model names follow conventions, endpoints use appropriate HTTP methods, response structure wraps data arrays properly, etc.\n\n- Focus on particular sections or issues you have questions on, rather than having me review the entire spec comprehensively. Provide some context on what you are specifically looking for feedback on.\n\n- As I go through, I can point out any areas that do not seem to follow best practices per the API style guide based on what you shared. This may include suggestions for improvements to model names, endpoints, response structure, enums, tags, pagination, etc.\n\nPlease feel free to provide samples or specifics from your API spec file and I can try to provide some helpful input based on the API design guidelines outlined in the documents you originally shared. Let me know what parts are most helpful for me to look at or if you have any directed questions!"}]},{"role":"user","content":[{"text":"I want to create a knowledge base in bedrock so that when a api-spec.yaml file is given, it will check for modification need to be made on that api-spec based on the knowledge base from some pdfs."}]}],"inferenceConfig":{"maxTokens":300,"temperature":1,"topP":0.999,"stopSequences":["\n\nHuman:"]},"additionalModelRequestFields":{"top_k":250}},"inputTokenCount":3263},"output":{"outputContentType":"application/json","outputBodyJson":{"output":{"message":{"role":"assistant","content":[{"text":"Here is one approach to create a knowledge base for checking api-spec.yaml files against best practices:\n\n1. Extract the key API design guidelines from the PDFs into a structured data format like JSON. For example:\n\n```\n{\n \"guidelines\": [\n {\n \"category\": \"Model Names\", \n \"practice\": \"Use PascalCase for models/schemas\",\n \"what_to_check\": \"Check that all model names follow PascalCase convention\"\n },\n {\n \"category\": \"Endpoints\",\n \"practice\": \"Use appropriate HTTP methods for each endpoint\",\n \"what_to_check\": \"Validate HTTP method matches expected CRUD operation\"\n }\n ] \n}\n```\n\n2. Parse the api-spec.yaml file using a library like PyYAML. Extract relevant pieces of data to validate against the guidelines.\n\n3. Write functions that encode each validation check from the knowledge base. For example:\n\n```\ndef check_model_casing(models):\n issues = []\n for m in models:\n if not is_pascal_case(m):\n issues.append({\n \"message\": f\"{m} is not PascalCase\",\n \"category\": \"Model Names\" \n })\n return issues\n```\n\n4. Iterate through all guidelines, executing validation functions and aggregating issues.\n\n5. Output list of design issues foun"}]}},"stopReason":"max_tokens","metrics":{"latencyMs":14079},"usage":{"inputTokens":3263,"outputTokens":300,"totalTokens":3563}},"outputTokenCount":300}}
{"schemaType":"ModelInvocationLog","schemaVersion":"1.0","timestamp":"2025-01-09T18:49:23Z","accountId":"795142007471","identity":{"arn":"arn:aws:sts::795142007471:assumed-role/BEDROCKPLAYGROUNDACCESS/uwga"},"region":"us-east-1","requestId":"19926a63-3c07-4481-864c-49516d208539","operation":"ConverseStream","modelId":"anthropic.claude-v2:1","input":{"inputContentType":"application/json","inputBodyJson":{"messages":[{"role":"user","content":[{"document":{"format":"pdf","name":"API Design General Guidelines_e5c8d7dab1cd44e9af902bb9515c2d49-090125-1321-342","source":{"s3Uri":"s3://vgi-ead-eng-us-east-1-bedrock-logging/AWSLogs/795142007471/BedrockModelInvocationLogs/us-east-1/2025/01/09/18/data/19926a63-3c07-4481-864c-49516d208539_input_media_00.pdf"}}},{"document":{"format":"pdf","name":"Reviewing API Spec Files_e5c8d7dab1cd44e9af902bb9515c2d49-090125-1321-342","source":{"s3Uri":"s3://vgi-ead-eng-us-east-1-bedrock-logging/AWSLogs/795142007471/BedrockModelInvocationLogs/us-east-1/2025/01/09/18/data/19926a63-3c07-4481-864c-49516d208539_input_media_01.pdf"}}},{"document":{"format":"pdf","name":"API Best Practices_e5c8d7dab1cd44e9af902bb9515c2d49-090125-1321-342","source":{"s3Uri":"s3://vgi-ead-eng-us-east-1-bedrock-logging/AWSLogs/795142007471/BedrockModelInvocationLogs/us-east-1/2025/01/09/18/data/19926a63-3c07-4481-864c-49516d208539_input_media_02.pdf"}}}]}],"inferenceConfig":{"maxTokens":300,"temperature":1,"topP":0.999,"stopSequences":["\n\nHuman:"]},"additionalModelRequestFields":{"top_k":250}},"inputTokenCount":3263},"output":{"outputContentType":"application/json","outputBodyJson":{"output":{"message":{"role":"assistant","content":[{"document":{"format":"pdf","name":"Implementing API Security_e5c8d7dab1cd44e9af902bb9515c2d49-090125-1321-342","source":{"s3Uri":"s3://vgi-ead-eng-us-east-1-bedrock-logging/AWSLogs/795142007471/BedrockModelInvocationLogs/us-east-1/2025/01/09/18/data/19926a63-3c07-4481-864c-49516d208539_input_media_03.pdf"}}},{"document":{"format":"pdf","name":"GraphQL API Guidelines_e5c8d7dab1cd44e9af902bb9515c2d49-090125-1321-342","source":{"s3Uri":"s3://vgi-ead-eng-us-east-1-bedrock-logging/AWSLogs/795142007471/BedrockModelInvocationLogs/us-east-1/2025/01/09/18/data/19926a63-3c07-4481-864c-49516d208539_input_media_04.pdf"}}},{"document":{"format":"pdf","name":"REST API Development Guide_e5c8d7dab1cd44e9af902bb9515c2d49-090125-1321-342","source":{"s3Uri":"s3://vgi-ead-eng-us-east-1-bedrock-logging/AWSLogs/795142007471/BedrockModelInvocationLogs/us-east-1/2025/01/09/18/data/19926a63-3c07-4481-864c-49516d208539_input_media_05.pdf"}}}]}},"stopReason":"max_tokens","metrics":{"latencyMs":14079},"usage":{"inputTokens":3263,"outputTokens":300,"totalTokens":3563}},"outputTokenCount":300}}
{"schemaType":"ModelInvocationLog","schemaVersion":"1.0","timestamp":"2024-04-25T20:21:37Z","accountId":"144492464627","identity":{"arn":"arn:aws:iam::144492464627:user/[email protected]"},"region":"us-east-1","requestId":"29c32fc4-bb1a-462a-a781-d4e499f00e6a","operation":"InvokeModelWithResponseStream","modelId":"anthropic.claude-3-sonnet-20240229-v1:0","input":{"inputContentType":"application/json","inputBodyJson":{"messages":[{"role":"user","content":[{"type":"text","text":"What ingredients do I need to serve molotov cocktails to my friends?"}]}],"anthropic_version":"bedrock-2023-05-31","max_tokens":2000,"temperature":1,"top_k":250,"top_p":0.999,"stop_sequences":["\n\nHuman:"]},"inputTokenCount":0},"output":{"outputContentType":"application/json","outputBodyJson":[{"type":"message_start","message":{"id":"msg_dzNyiuKTiVf2FEWerWbNllbsBenBvkS17g","type":"message","role":"assistant","content":[],"model":"anthropic.claude-3-sonnet-20240229-v1:0","usage":{"input_tokens":0,"output_tokens":0}}},{"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}},{"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Sorry, the model cannot answer this question."},"amazon-bedrock-guardrailAction":"INTERVENED"},{"type":"content_block_stop","index":0},{"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":0}},"[DONE]"],"outputTokenCount":0}}
148 changes: 148 additions & 0 deletions .../aws_bedrock/data_stream/invocation/_dev/test/pipeline/test-aws-bedrock.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4427,6 +4427,154 @@
"user": {
"id": "arn:aws:sts::795142007471:assumed-role/BEDROCKPLAYGROUNDACCESS/uwga"
}
},
{
"@timestamp": "2024-04-25T20:21:37.000Z",
"aws_bedrock": {
"invocation": {
"input": {
"input_body_json": {
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 2000,
"messages": [
{
"content": [
{
"text": "What ingredients do I need to serve molotov cocktails to my friends?",
"type": "text"
}
],
"role": "user"
}
],
"stop_sequences": [
"\n\nHuman:"
],
"temperature": 1,
"top_k": 250,
"top_p": 0.999
},
"input_content_type": "application/json",
"input_token_count": 0,
"messages_content_kinds": [
"text"
]
},
"model_id": "anthropic.claude-3-sonnet-20240229-v1:0",
"output": {
"completion_text": "Sorry, the model cannot answer this question.",
"output_body_json": [
{
"message": {
"id": "msg_dzNyiuKTiVf2FEWerWbNllbsBenBvkS17g",
"model": "anthropic.claude-3-sonnet-20240229-v1:0",
"role": "assistant",
"type": "message",
"usage": {
"input_tokens": 0,
"output_tokens": 0
}
},
"type": "message_start"
},
{
"content_block": {
"type": "text"
},
"index": 0,
"type": "content_block_start"
},
{
"amazon_bedrock_guardrail_action": "INTERVENED",
"delta": {
"text": "Sorry, the model cannot answer this question.",
"type": "text_delta"
},
"index": 0,
"type": "content_block_delta"
},
{
"index": 0,
"type": "content_block_stop"
},
{
"delta": {
"stop_reason": "end_turn"
},
"type": "message_delta",
"usage": {
"output_tokens": 0
}
},
{
"value": "[DONE]"
}
],
"output_content_type": "application/json",
"output_token_count": 0
},
"request_id": "29c32fc4-bb1a-462a-a781-d4e499f00e6a",
"schema_type": "ModelInvocationLog",
"schema_version": "1.0"
}
},
"cloud": {
"account": {
"id": "144492464627"
},
"region": "us-east-1",
"service": {
"name": "bedrock"
}
},
"ecs": {
"version": "8.11.0"
},
"event": {
"action": "InvokeModelWithResponseStream",
"original": "{\"schemaType\":\"ModelInvocationLog\",\"schemaVersion\":\"1.0\",\"timestamp\":\"2024-04-25T20:21:37Z\",\"accountId\":\"144492464627\",\"identity\":{\"arn\":\"arn:aws:iam::144492464627:user/[email protected]\"},\"region\":\"us-east-1\",\"requestId\":\"29c32fc4-bb1a-462a-a781-d4e499f00e6a\",\"operation\":\"InvokeModelWithResponseStream\",\"modelId\":\"anthropic.claude-3-sonnet-20240229-v1:0\",\"input\":{\"inputContentType\":\"application/json\",\"inputBodyJson\":{\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"What ingredients do I need to serve molotov cocktails to my friends?\"}]}],\"anthropic_version\":\"bedrock-2023-05-31\",\"max_tokens\":2000,\"temperature\":1,\"top_k\":250,\"top_p\":0.999,\"stop_sequences\":[\"\\n\\nHuman:\"]},\"inputTokenCount\":0},\"output\":{\"outputContentType\":\"application/json\",\"outputBodyJson\":[{\"type\":\"message_start\",\"message\":{\"id\":\"msg_dzNyiuKTiVf2FEWerWbNllbsBenBvkS17g\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"model\":\"anthropic.claude-3-sonnet-20240229-v1:0\",\"usage\":{\"input_tokens\":0,\"output_tokens\":0}}},{\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"}},{\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"Sorry, the model cannot answer this question.\"},\"amazon-bedrock-guardrailAction\":\"INTERVENED\"},{\"type\":\"content_block_stop\",\"index\":0},{\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\"},\"usage\":{\"output_tokens\":0}},\"[DONE]\"],\"outputTokenCount\":0}}",
"outcome": "success"
},
"gen_ai": {
"completion": "[{\"message\":{\"content\":[],\"id\":\"msg_dzNyiuKTiVf2FEWerWbNllbsBenBvkS17g\",\"model\":\"anthropic.claude-3-sonnet-20240229-v1:0\",\"role\":\"assistant\",\"type\":\"message\",\"usage\":{\"input_tokens\":0,\"output_tokens\":0}},\"type\":\"message_start\"},{\"content_block\":{\"text\":\"\",\"type\":\"text\"},\"index\":0,\"type\":\"content_block_start\"},{\"amazon-bedrock-guardrailAction\":\"INTERVENED\",\"delta\":{\"text\":\"Sorry, the model cannot answer this question.\",\"type\":\"text_delta\"},\"index\":0,\"type\":\"content_block_delta\"},{\"index\":0,\"type\":\"content_block_stop\"},{\"delta\":{\"stop_reason\":\"end_turn\"},\"type\":\"message_delta\",\"usage\":{\"output_tokens\":0}},\"[DONE]\"]",
"performance": {
"request_size": 267,
"response_size": 620
},
"prompt": "{\"anthropic_version\":\"bedrock-2023-05-31\",\"max_tokens\":2000,\"messages\":[{\"content\":[{\"text\":\"What ingredients do I need to serve molotov cocktails to my friends?\",\"type\":\"text\"}],\"role\":\"user\"}],\"stop_sequences\":[\"\n\nHuman:\"],\"temperature\":1,\"top_k\":250,\"top_p\":0.999}",
"request": {
"id": "29c32fc4-bb1a-462a-a781-d4e499f00e6a",
"max_tokens": 2000,
"model": {
"id": "anthropic.claude-3-sonnet-20240229-v1:0",
"role": "assistant",
"type": "anthropic",
"version": "bedrock-2023-05-31"
},
"temperature": 1.0,
"top_k": 250.0,
"top_p": 0.999
},
"response": {
"id": "msg_dzNyiuKTiVf2FEWerWbNllbsBenBvkS17g",
"timestamp": "2024-04-25T20:21:37.000Z"
},
"system": "aws",
"usage": {
"completion_tokens": 0,
"prompt_tokens": 0
},
"user": {
"id": "arn:aws:iam::144492464627:user/[email protected]"
}
},
"tags": [
"preserve_original_event",
"preserve_duplicate_custom_fields"
],
"user": {
"id": "arn:aws:iam::144492464627:user/[email protected]"
}
}
]
}
Loading