Feature: Improve lenient parsing: audit compat coverage and expand test cases

[allenporter]

Summary

Audit the existing ical.compat compatibility layer, document what real-world broken ICS patterns are and are not handled, and expand test coverage with real-world broken calendar file examples.

What We Already Handle

The ical.compat module has a well-designed, PRODID-aware compatibility layer that auto-detects the calendar producer and enables targeted fixups:

Microsoft Exchange Server (PRODID contains "Microsoft Exchange Server")

• Invalid timezones (enable_allow_invalid_timezones) — tolerates TZID references that don't match any VTIMEZONE definition
• Extended timezone names (enable_extended_timezones) — handles non-IANA timezone identifiers like "Eastern Standard Time" that Windows/Exchange uses instead of "America/New_York"

Google Calendar (PRODID contains "Google Inc//Google Calendar")

• DTSTART/UNTIL type mismatch (enable_dtstart_until_compat) — Google Calendar generates RRULE UNTIL as a DATE when DTSTART is DATE-TIME, violating RFC 5545 section 3.3.10. The compat layer silently converts UNTIL to match DTSTART's type.

All Producers (always enabled in compat mode)

• Same-day DTEND (enable_same_day_dtend_compat) — some providers set DTSTART and DTEND to the same date for all-day events; RFC 5545 requires DTEND to be the next day (non-inclusive)
• Invalid date formats (enable_allow_invalid_dates) — tolerates DATE properties with time suffixes (e.g., 20240115T000000 where 20240115 is expected)

Design Pattern

The compat layer uses a context manager pattern (enable_compat_mode()) with contextvars for thread-safe state. This is an opt-in API — callers must explicitly wrap their parsing calls. This is a deliberate design choice that keeps the default parsing strict/correct.

What Developers Complain About (Real-World Broken ICS Patterns)

Based on research of Stack Overflow, GitHub issues across iCal libraries, Reddit threads, and tools like ics-fixer and x-wr-timezone:

High Frequency — Most Commonly Reported

1. X-WR-TIMEZONE without VTIMEZONE blocks — Google Calendar exports sometimes include only a calendar-level X-WR-TIMEZONE header and omit per-event VTIMEZONE definitions entirely. The separate x-wr-timezone library exists solely to normalize this. Status in ical: ❓ Not handled — this is likely the biggest remaining gap.
2. Missing VTIMEZONE with valid IANA TZID — Many generators include TZID=America/New_York but skip the VTIMEZONE block, assuming the client knows standard IANA zones. Technically a spec violation, but trivially resolvable since we have the tzdata package. Status: ❓ Partially handled via Exchange compat, but should work for all producers.

Moderate Frequency

3. Improper line folding — RFC 5545 requires lines >75 octets to be folded (newline + space). Many generators produce long unfoldedlines, especially in DESCRIPTION and LOCATION fields. Status: ❓ Needs investigation — pyparsing grammar may already handle this.
4. Missing UID on events — Some generators omit the UID property entirely. This causes issues for recurrence and deduplication. Status: ❓ Needs investigation.
5. Smart quotes and encoding issues — Descriptions copy-pasted from Word/rich-text contain smart quotes (U+201C/U+201D), em-dashes (U+2014), etc. These may cause issues if not properly UTF-8 encoded. Status: ❓ Needs investigation.

Lower Frequency

6. Malformed ORGANIZER/ATTENDEE — Empty values, missing mailto: prefix, bare email addresses. Common from Exchange and older GroupWise systems. Status: ❓ Needs investigation.
7. DURATION vs DTEND conflicts — Both specified on the same event, which is explicitly forbidden by RFC 5545. Status: ❓ Needs investigation.

Proposed Actions

1. Investigate and document current behavior

For each pattern above, create a test .ics file that exercises the broken pattern and document whether ical handles it (with and without compat mode).

2. Expand test coverage

Add test cases in tests/compat/ with real-world broken ICS samples. Each test should:

• Parse the broken ICS without compat mode and verify it either fails gracefully or succeeds
• Parse the broken ICS with enable_compat_mode() and verify it produces correct results
• Document which calendar producer generates this pattern

3. Priority fixes

Based on frequency, the highest-value fixes are:

1. X-WR-TIMEZONE handling (covers a huge class of Google Calendar exports)
2. Missing VTIMEZONE with valid IANA TZID for all producers (not just Exchange)
3. Long line handling (if not already supported)

4. Documentation

The compat layer is a genuinely great feature that's currently invisible. Document it prominently:

• Add a "Handling broken ICS files" section to the docs
• Show the enable_compat_mode() context manager pattern in the README or quickstart

Contributions Welcome

If you encounter a broken ICS file that ical doesn't handle, please open an issue with:

1. The (sanitized) ICS content
2. What calendar application produced it
3. What you expected to happen vs. what actually happened

[proscar87]

Ran the audit from "Proposed Actions" step 1 — a case per pattern, parsed with and without compat mode, checking the resulting value rather than just that parsing didn't raise. Results below; three of the seven statuses in the description turn out to be wrong.

#	Pattern	Documented status	Measured
1	X-WR-TIMEZONE without VTIMEZONE	❓ "likely the biggest remaining gap"	✅ handled
2	Missing VTIMEZONE, valid IANA TZID	❓ partially, Exchange only	✅ handled for all producers
3	Unfolded long lines	❓ needs investigation	✅ handled
4	Missing UID	❓ needs investigation	✅ handled
5	Smart quotes / em dashes	❓ needs investigation	✅ handled
6	ORGANIZER without mailto:	❓ needs investigation	✅ parses
7	DURATION + DTEND	❓ needs investigation	✅ fixed in #669

1 — X-WR-TIMEZONE is applied. A floating DTSTART:20250715T140000 under X-WR-TIMEZONE:America/New_York resolves to ZoneInfo('America/New_York'), 18:00Z. This is the item the description calls the biggest remaining gap and cites the x-wr-timezone library for. It works, in strict mode, with no compat needed.

2 — works for every producer, not just Exchange. DTSTART;TZID=America/New_York with no VTIMEZONE block resolves to America/New_York under a generic PRODID. The description proposes "make this work for all producers, not just Exchange" as priority 2; that's already the behaviour.

3 — both directions. A 200-character unfolded SUMMARY comes back at 200 characters, and a properly folded line unfolds to 'parte unoparte dos' — correct per rfc5545 §3.1, where the CRLF and the single leading space are both removed.

4 — a UUID is generated. A missing UID yields e.g. 79d635f0-9858-11f1-ad71-9edfaeb6fa23. Worth noting the consequence rather than just the mechanism: it is not stable across parses, so deduplicating the same file read twice will not match on uid. That may be the right trade or may deserve a note in the docs — flagging rather than proposing.

5 and 6 — no data loss. “Hola” — adios round-trips byte for byte. ORGANIZER:oscar@example.com parses to a CalAddress with uri='oscar@example.com'; the missing mailto: is preserved rather than rejected or silently rewritten, which seems the right call for a lenient parser.

---

So the remaining work here is smaller than the description suggests. What I did not find is a case that fails, which means I can't propose a fix without inventing one.

Two things that would move it forward, neither of which I can do alone:

• Real broken files. The description asks for real-world samples in tests/compat/; my cases are synthetic, written from the descriptions. A synthetic file exercises what I imagined the pattern to be, which is exactly how a pattern gets marked "handled" while the real thing still breaks.
• Correcting the description, so items 1 and 2 don't stay on the priority list. Happy to do that if you'd rather have it edited than sitting in a comment.

Measured against main at 5aca642, both strict and enable_compat_mode().

🤖 Generated with Claude Code