[Microsoft Teams] Rewrite connector to use application-only permissions

[Jan-Kazlouski-elastic]

Closes #4145

Rewrites the Microsoft Teams connector to application-only authentication (client secret or certificate), matching SharePoint Online / Outlook. Removes the legacy dual auth model (username/password + delegated Graph) that cannot work headlessly with MFA.

Auth and permissions

• Application permissions only (no user sign-in, no Teams app install, no RSC / WhereInstalled).
• Tenant-wide Graph .All application permissions, documented in microsoft_teams/README.md:
  • Team.ReadBasic.All, TeamMember.Read.All
  • User.ReadBasic.All (profile hydrate for User docs + DLS identity email: / user: / user_id:)
  • Channel.ReadBasic.All, ChannelMember.Read.All, ChannelMessage.Read.All
  • Chat.ReadBasic.All, Chat.Read.All
  • Files.Read.All when "Fetch attachment content" is enabled (validated from the app token roles claim in validate_config)
• Chat discovery walks team members → GET /users/{id}/chats (deduped by chat id). There is no tenant-wide GET /users or /chats sync.
• ChannelMessage.Read.All / Chat.Read.All remain Microsoft protected Teams APIs; admin consent alone may not be enough until protected API access is granted for the app.

Why not RSC / least-privilege WhereInstalled (issue 3.1): RSC only scopes message reads after a Teams app is installed into each team/chat. There is no tenant-wide "list chats where installed" path for a crawler; install UX/Graph packaging failed in practice for an RSC-only (non-bot) package; requiring install-everywhere is not operable for enterprise search. Privacy for end users is enforced with DLS from membership (same model as SPO/Outlook), not by limiting what the crawler app can read.

Shared Graph auth/HTTP (connectors/sources/shared/microsoft/graph.py) is reused by SharePoint Online.

Sync model

teams → team members (dedupe user ids) → resolve profiles via GET /users/{id} (batched)
                ↓                              ↓
         Users (content) + identity docs    chats via GET /users/{id}/chats
         team / channels / messages         GET /chats/{id}/members (full ACL)
         channel Files folder (optional)    chat messages (+ attachments if enabled)
         GET .../messages/{id}/replies

• One in-memory pass per sync for user discovery (no second team/member walk for chats).
• Content types: Team, Channel, Channel Message, Chat, Chat Message, User, File.
• Team / Channel / Chat include member_ids (Entra user ids). User docs include name, email (Graph mail only), upn (Graph userPrincipalName).
• Files (when "Fetch attachment content" is on): channel Files-folder drive items plus message reference attachments resolved via contentUrl → shares API. Each driveItem id is queued once per sync (folder + message rediscovery merges parents/ACLs in memory; messages link via attachments: [{id, title}]).
• Chat DLS ACLs always from dedicated GET /chats/{id}/members (not $expand=members, which Graph caps at 25).
• Channel replies always from dedicated GET .../messages/{id}/replies (not incomplete $expand=replies).
• Standard channels inherit team ACL; private/shared channels resolve channel members and fail closed (skip that channel) if members cannot be resolved under DLS.
• Content _allow_access_control uses user_id: only (Entra userId, never conversationMember id). Identity docs carry user_id: + email: (mail) + user: (UPN). User content docs omit _allow_access_control (directory metadata).
• PermissionsMissing (401/403) fails content sync and ACL sync. NotFound (404) may soft-skip absent resources and is summarized at end of sync.
• Enumeration refuses a successful sync when team and/or chat corpus enumeration fails with permissions errors (avoids wiping previously indexed content).
• Concurrent enumeration + consume (fixes prior stall/deadlock on large tenants).
• Extraction service HTTP session is closed on sync end when "Use text extraction service" is enabled.

Upgrade / corpus change (callout)

On first successful full sync after upgrade, documents that this rewrite no longer produces are deleted from the content index if they were left by the previous connector, including: calendars, channel/chat tabs, meeting recordings, and legacy shapes such as Team Member (replaced by User). Operators should expect that cleanup. Synced going forward: teams, channels, channel messages/replies, chats of team members, chat messages, Users (team members), Files (when enabled), and DLS identities.

Tests

• Unit coverage for discovery, dedicated members/replies, fail-loud permissions, Files role check, DLS (user_id:-only content ACL; identity dialects), File single-queue / deferred _id, extraction session close, membership-id ACL ignore.
• Fixture updated for /users/{id}/chats, /chats/{id}/members, message replies, and user profile routes.
• Manual E2E against a real tenant (content + ACL sync + text extraction for Files).

Checklists

Pre-Review Checklist

• this PR does NOT contain credentials of any kind, such as API keys or username/passwords (double check config.yml.example)
• this PR has a meaningful title
• this PR links to all relevant github issues that it fixes or partially addresses
• if there is no GH issue, please create it. Each PR should have a link to an issue
• this PR has a thorough description
• Covered the changes with automated tests
• Tested the changes locally
• Added a label for each target release version (example: v7.13.2, v7.14.0, v8.0.0)
• For bugfixes: backport safely to all minor branches still receiving patch releases
• Considered corresponding documentation changes
• Contributed any configuration settings changes to the configuration reference
• if you added or changed Rich Configurable Fields for a Native Connector, you made a corresponding PR in Kibana

Changes Requiring Extra Attention

• Security-related changes (encryption, TLS, SSRF, etc)
• New external service dependencies added.

Related Pull Requests

• N/A (Kibana native-connector RCF PR still needed if/when this ships as a native connector schema change: User, member_ids, upn, Files).

Release Note

Rewrote the Microsoft Teams connector for application-only authentication (client secret or certificate) using tenant-wide Graph application permissions and document-level security, replacing username/password + delegated auth. Syncs Teams, Channels, Chats (of team members), Users, messages/replies, and Files; content DLS uses user_id: membership tokens with identity docs for email/UPN login. Upgrading removes previously indexed calendars, tabs, meeting recordings, and legacy Team Member documents on the next successful full sync.

[erikcurrin-elastic]

On first successful full sync after upgrade, documents that this rewrite no longer produces are deleted from the content index if they were left by the previous connector, including: calendars, channel/chat tabs, meeting recordings, and legacy shapes such as Team Member (replaced by User). Operators should expect that cleanup. Synced going forward: teams, channels, channel messages/replies, chats of team members, chat messages, Users (team members), Files (when enabled), and DLS identities.

Calendar sync could still be nice, so not sure why this dropped it? It would require Calendar.Read

Meeting recordings - are we just relying on Sharepoint if the user wants this? Otherwise, why drop them?

Channel and call metadata, again might be valuable.

[Jan-Kazlouski-elastic]

Hi @erikcurrin-elastic

According to #4145 Teams connector must focus on teams native target entities (teams, channels, chats, messages, members, attachments). For all the entities we've dropped compared to legacy connector - main reason is that they are not listed in the tracking issue.

Calendars

Outlook connector seems to be a better fit for calendars today. Teams connector would be fetching only a subset of the same entities that are fetched by Outlook connector.

Meeting recordings

The legacy connector did not index recording binary/transcripts. It emitted lightweight metadata docs when a channel/chat system message carried callRecordingEventMessageDetail and the URL pointed at SharePoint (title + recording_url).
So, SharePoint connector seems a better fit for meeting recordings.

Channel / call metadata (tabs, call-ended, etc.)

Being a native MS Teams entities - these could be synced, though I am not sure if they are very useful. We could take it as a follow up, or include it here.

[Jan-Kazlouski-elastic]

Microsoft Teams connector — Elasticsearch document schemas

Graph resources: team, channel, chat, chatMessage, conversationMember, user, driveItem.

DLS: _allow_access_control is stamped on Team, Channel, Chat, Channel Message, Chat Message, and File when use_document_level_security is on. Values are user_id:{Entra oid} from conversationMember.userId. User content docs and ACL identity docs are separate (see below).

Skipped messages: deleted (deletedDateTime), messageType containing unknownFutureValue, or empty subject + body + attachments.

---

Team

GET /teams, enriched with GET /teams/{team-id} when webUrl / createdDateTime missing.

Elasticsearch field	Graph field	Summary
_id	id	Team id (Microsoft 365 group id).
type	—	Constant "Team".
title	displayName	Team name.
description	description	Team description.
url	webUrl	Teams web URL (often null on list; enriched via single-team GET).
creation_time	createdDateTime	Team creation time (enriched when missing from list).
_timestamp	—	Indexing time (iso_zulu() at emit).
member_ids	conversationMember[].userId	Sorted unique Entra user ids from GET /teams/{team-id}/members.
_allow_access_control	conversationMember[].userId	DLS tokens user_id:{userId} for all team members.

---

Channel

GET /teams/{team-id}/channels

Elasticsearch field	Graph field	Summary
_id	id	Channel id.
type	—	Constant "Channel".
title	displayName	Channel name.
description	description	Channel description.
url	webUrl	Channel web URL.
creation_time	createdDateTime	Channel creation time.
_timestamp	—	Indexing time at emit.
team_id	—	Parent team id (from producer context, not on channel resource).
team_title	—	Parent team displayName.
member_ids	conversationMember[].userId	Standard channels: same as parent team members. Private/shared (membershipType private / shared): from GET /teams/{team-id}/channels/{channel-id}/members.
_allow_access_control	conversationMember[].userId	Standard: team ACL. Private/shared: channel-member ACL (user_id:{userId}). Channel skipped entirely if private/shared members cannot be resolved and DLS is on.

---

Channel Message

GET /teams/{team-id}/channels/{channel-id}/messages and GET .../messages/{message-id}/replies

Elasticsearch field	Graph field	Summary
_id	id	Message id.
type	—	Constant "Channel Message".
url	webUrl	Message web URL (if Graph provides it).
_timestamp	lastModifiedDateTime	Last modified time.
creation_time	createdDateTime	Message creation time.
channel_id	—	Parent channel id (producer context).
channel_title	—	Parent channel displayName.
sender_name	from.user.displayName	Sender display name (empty if missing).
sender_id	from.user.id	Sender Entra user id.
subject	subject	Plaintext subject / file-share caption (trimmed).
message	body.content	HTML body converted to plain text (html_to_text).
reply_to_id	replyToId	Parent message id for thread replies; empty string if root.
attachments	attachments[] (reference only)	[{id, title}] linking to File docs. Only contentType == "reference" with resolvable contentUrl → GET /shares/{encoded}/driveItem.
_allow_access_control	—	Inherited from parent channel ACL (team or channel members).

---

Chat

Discovered via GET /users/{user-id}/chats (deduped by chat id); members via GET /chats/{chat-id}/members.

Elasticsearch field	Graph field	Summary
_id	id	Chat id.
type	—	Constant "Chat".
title	topic	Chat topic; if empty, comma-joined conversationMember.displayName list.
url	webUrl	Chat web URL.
chatType	chatType	Graph chat type (oneOnOne, group, meeting, etc.).
_timestamp	lastUpdatedDateTime	Last updated time.
creation_time	createdDateTime	Chat creation time.
member_ids	conversationMember[].userId	Sorted unique Entra user ids from chat members API.
_allow_access_control	conversationMember[].userId	DLS tokens user_id:{userId} for all chat members.

---

Chat Message

GET /chats/{chat-id}/messages

Elasticsearch field	Graph field	Summary
_id	id	Message id.
type	—	Constant "Chat Message".
url	webUrl	Message URL; falls back to parent chat webUrl if message has none.
_timestamp	lastModifiedDateTime	Last modified time.
creation_time	createdDateTime	Message creation time.
chat_id	chat.id	Parent chat id.
chat_title	chat.topic or members	Chat topic, or comma-joined member display names if topic empty.
sender_name	from.user.displayName	Sender display name.
sender_id	from.user.id	Sender Entra user id.
subject	subject	Plaintext subject (trimmed).
message	body.content	HTML body → plain text.
attachments	attachments[] (reference only)	Same as Channel Message: [{id, title}] → File docs via shares API.
_allow_access_control	—	Inherited from parent chat member ACL.

---

User

Content document (directory metadata). No _allow_access_control. One doc per Teams participant (team ∪ channel ∪ chat members).

Elasticsearch field	Graph field	Summary
_id	id	Entra user id (oid).
type	—	Constant "User".
name	displayName	From GET /users/{id}?$select=id,mail,userPrincipalName,displayName; falls back to conversationMember.displayName on 404.
email	mail	Mailbox only (never UPN). Empty if mail null.
upn	userPrincipalName	Entra login / UPN.
_timestamp	—	Indexing time at emit.

---

File

When fetch_attachment_content is on. Sources: (1) channel Files folder GET /teams/{team-id}/channels/{channel-id}/filesFolder → recursive GET /drives/{drive-id}/items/{item-id}/children, (2) message reference attachments via GET /shares/{encoded-contentUrl}/driveItem. One ES doc per driveItem.id per sync (deduped).

Elasticsearch field	Graph field	Summary
_id	id	Drive item id.
type	—	Constant "File".
title	name	File name.
url	webUrl	File web URL.
size_in_bytes	size	File size in bytes.
_timestamp	lastModifiedDateTime	Last modified time.
creation_time	createdDateTime	Created time.
channel_id	—	Set when discovered from channel Files folder or channel message attachment (sparse; omitted if unknown).
channel_title	—	Parent channel displayName when known.
chat_id	—	Set when discovered from chat message attachment (sparse).
chat_title	—	Chat topic or member names when known.
body	—	Extracted text when use_text_extraction_service is on (GET /drives/{drive-id}/items/{item-id}/content → extraction service).
_attachment	—	Base64 file content for ingest pipeline when text extraction service is off.
_allow_access_control	—	Union of ACLs from all discovery paths that referenced this driveItem (channel and/or chat context).

---

Access Control (identity document)

Emitted by get_access_control() into the ACL index (not the main content index). Only when DLS is enabled.

Elasticsearch field	Graph field	Summary
_id	id	Entra user id (same as User content doc).
identity.user_id	id	Prefixed user_id:{oid}.
identity.email	mail	Prefixed email:{mail} if mailbox present.
identity.user	userPrincipalName	Prefixed user:{upn} if UPN present (SPO-aligned dialect).
created_at	—	ACL doc creation time (iso_zulu()).
query	—	Elasticsearch DLS query template matching _allow_access_control.keyword on content docs.

---

Message attachments nested shape

Not raw Graph attachments — connector-resolved references only:

Elasticsearch field	Graph field	Summary
attachments[].id	driveItem.id	File document _id (from shares resolution of attachments[].contentUrl).
attachments[].title	driveItem.name or attachments[].name	Display title; falls back to Graph attachment name.