`convert` date-time results depend on the JVM default time zone

[zaleslaw]

Summary

convert's date-time conversions resolve zone-dependent steps against TimeZone.currentSystemDefault(). The same
operation on the same DataFrame produces different values depending on the machine it runs on, and for several
conversions there is no way to pin the zone.

core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/api/convert.kt:837:

internal val defaultTimeZone get() = TimeZone.currentSystemDefault()

58 entries of the converter table (impl/api/convert.kt:374–:800) read it. On the public side, 24 overloads in
api/convert.kt accept zone: TimeZone = defaultTimeZone — but they cover only the receivers Long, Int,
kotlin.time.Instant and kotlinx.datetime.Instant, and only the direction towards
LocalDate/LocalTime/LocalDateTime. Everything else takes the ambient zone with no opt-out.

All values below are measured on the current master (JDK 21), not derived by hand.

Why it matters

• Results are not reproducible across environments. A pipeline that converts an epoch column to
  LocalDateTime and then groups, filters, asserts, or persists produces different output on a developer machine
  and on CI.
• The escape hatch is incomplete and its availability depends on how the column was selected, not on the data
  (case 3, case 5).
• The library already promises the opposite elsewhere. docs/StardustDocs/topics/dataSources/ApacheArrow.md
  and Parquet.md state that timestamp writing resolves against UTC and never against the default zone, while
  docs/StardustDocs/topics/convert.md does not mention time zones at all — the string TimeZone does not appear
  in it.
• A downstream module already had to work around it. dataframe-arrow reimplemented two of core's
  converters privately (convertToInstantInUtc / convertToLocalDateTimeInUtc in
  dataframe-arrow/.../io/ArrowWriterImpl.kt) because the same DataFrame was writing different Arrow/Parquet
  bytes on different machines (Support Parquet UTC timestamp vectors (TimeStampXXXTZVector) in Arrow reader #926).

How it works today

Three properties together mean the zone cannot reach the converter table at all:

internal typealias TypeConverter = (Any) -> Any?	impl/api/convert.kt:182 — a converter's only input is the cell value
mutableMapOf<Triple<KType, KType, ParserOptions?>, TypeConverter?>	impl/api/convert.kt:171 — the converter cache key has no zone, and ParserOptions carries none
defaultTimeZone is a get() property	impl/api/convert.kt:837 — re-read on every call, from inside the per-cell lambda

Dispatch is reflective on jvmErasure in createConverter (impl/api/convert.kt:199), keyed on the column's
runtime KType. Any zone-aware fix has to change TypeConverter's shape or thread a zone through
convertToTypeImpl / getConverter / createConverter.

Cases

1. The same epoch value becomes a different wall-clock reading per machine

val millis = 1_704_110_400_123L // 2024-01-01T12:00:00.123Z
columnOf(millis).convertToLocalDateTime()[0]

Default zone	Result
UTC	2024-01-01T12:00:00.123
Europe/Berlin	2024-01-01T13:00:00.123
Pacific/Kiritimati	2024-01-02T02:00:00.123
America/Los_Angeles	2024-01-01T04:00:00.123

Note what is not broken: Long → LocalDateTime → Long does round-trip to 1704110400123 in all four zones,
because both halves read the same ambient zone. The problem is the intermediate value — it is what gets grouped,
compared, asserted and written out — and the round trip only holds while both halves run in the same environment.

For contrast, Long → Instant is zone-free and gives 2024-01-01T12:00:00.123Z in all four zones
(impl/api/convert.kt:517, Instant.fromEpochMilliseconds), while Long → LocalDateTime at :511 is not. Two
adjacent rows of the same table, two different contracts.

2. LocalDate has two different meanings of "start of day", nine lines apart

val date = LocalDate(2024, 1, 1)
columnOf(date).convertTo<LocalDateTime>()[0] // zone-free:  impl/api/convert.kt:758, atTime(0, 0)
columnOf(date).convertTo<Instant>()[0]       // ambient:    impl/api/convert.kt:760, atStartOfDayIn(defaultTimeZone)
columnOf(date).convertToLong()[0]            // ambient:    impl/api/convert.kt:767

Default zone	→ LocalDateTime	→ Instant	→ Long
UTC	2024-01-01T00:00	2024-01-01T00:00:00Z	1704067200000
Europe/Berlin	2024-01-01T00:00	2023-12-31T23:00:00Z	1704063600000
Pacific/Kiritimati	2024-01-01T00:00	2023-12-31T10:00:00Z	1704016800000
America/Los_Angeles	2024-01-01T00:00	2024-01-01T08:00:00Z	1704096000000

A LocalDate of January 1st becomes December 31st on two of the four machines. There is no zone parameter on
either of the two ambient conversions.

3. Whether you can pin the zone depends on how you selected the column

val df = dataFrameOf("ts")(1_704_110_400_123L)

df.convert("ts").toLocalDateTime()          // Convert<T, Any?>  -> api/convert.kt:2315, no zone
df.convert { ts }.toLocalDateTime(zone)     // Convert<T, Long?> -> api/convert.kt:2165, zone honoured

convert(vararg columns: String) returns Convert<T, Any?> (api/convert.kt:295). Convert<T, out C> is
covariant, but Any? is not a subtype of Long?, so the zone-aware overload is inapplicable and resolution falls
to the zone-free Convert<T, *>.toLocalDateTime(). Measured under Europe/Berlin, the first line yields
2024-01-01T13:00:00.123.

So string-selected and typed-accessor users get different determinism guarantees for the identical column.

4. An erased column has no way out at all

For col: AnyCol (= DataColumn<*>), the only applicable overload is
DataColumn<T?>.convertToLocalDateTime() (api/convert.kt:702), which takes no zone. There is no
DataColumn<*>.convertToLocalDateTime(zone) anywhere. Generic code that receives a column cannot be made
deterministic without changing the JVM default zone around the call.

5. Conversions with no zone parameter anywhere in the public API

val inst = Instant.parse("2024-01-01T23:30:00Z")
columnOf(inst).convertToLocalDate()[0]

Default zone	→ LocalDate	→ LocalTime
UTC	2024-01-01	23:30
Europe/Berlin	2024-01-02	00:30
Pacific/Kiritimati	2024-01-02	13:30
America/Los_Angeles	2024-01-01	15:30

convertToLocalDate / convertToLocalTime have zone overloads only for Long and Int receivers — there is no
DataColumn<Instant>.convertToLocalDate(zone), even though Instant → LocalDateTime does get one
(api/convert.kt:2008). The whole reverse direction is affected too: LocalDateTime/LocalDate →
Instant/Long/java.time.Instant (impl/api/convert.kt:706, :713, :721, :760, :767, :773) has no
zone-aware entry point at all. Same for Byte/Short sources and for every java.time.* source.

For these, the only workaround is mutating java.util.TimeZone.setDefault around the call.

6. DST gaps break the value silently, and only on some machines

val gap = LocalDateTime(2024, 3, 31, 2, 30) // does not exist in Europe/Berlin
columnOf(gap).convertTo<Instant>()[0]

Default zone	→ Instant	back to LocalDateTime	round-trip equal
UTC	2024-03-31T02:30:00Z	2024-03-31T02:30	yes
Europe/Berlin	2024-03-31T01:30:00Z	2024-03-31T03:30	no

The local time 02:30 does not exist in Berlin on that date, so it is shifted forward by the gap and the round
trip returns 03:30. The same code on a UTC machine is lossless.

The overlap case (2024-10-27T02:30, which happens twice in Berlin) round-trips equal but silently picks the
earlier of the two valid instants (2024-10-27T00:30:00Z) — a one-hour ambiguity with no signal to the caller.

7. defaultTimeZone is read per value, not per column

Because defaultTimeZone is a get() property read inside the per-cell lambda, a cached converter does not
freeze the zone. A default-zone change during a long-running conversion would produce a column with mixed
interpretations rather than a consistently stale one. This is a property of the mechanism; I did not manage to
produce an observable failure from it, so it is listed as a robustness note rather than a reproducible bug.

Current test coverage

Worth stating plainly, because it cuts both ways: there is no value-level test pinning this behaviour in either
direction. The current behaviour is unprotected, and a change of the default would be invisible to CI.

• core/src/test/kotlin/.../api/convert.kt — the test Instant to LocalDateTime calls
  df.convert { time }.toLocalDateTime() on a Clock.System.now() column and asserts nothing at all.
• core/src/test/kotlin/.../io/ParserTests.kt, convert to date and time — the only test of the zone-taking
  overloads. It passes TimeZone.UTC everywhere and never a second zone, so it would still pass if the zone
  argument were ignored.
• samples/src/test/kotlin/.../api/Convert.kt — the largest cluster of zone-dependent conversions; every
  assertion checks type and nullability only, never a value.
• TimeZone.setDefault appears nowhere under core/src/test. The only helper that runs a block under a fixed
  default zone is a private withDefaultTimeZone in dataframe-arrow's ArrowTimestampTzTest; there is no
  shared one in common-test-utils.

Proposed tests

House style is JUnit 4 + kotest matchers, backticked sentence names, one test per claim (TEST_GUIDELINES.md).
Natural home: core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/api/convert.kt (ConvertTests).

A shared helper is needed first — worth promoting into common-test-utils rather than copying a third time:

/** Runs [block] with the JVM default time zone set to [zoneId], restoring the previous one afterwards. */
fun <R> withDefaultTimeZone(zoneId: String, block: () -> R): R {
    val previous = java.util.TimeZone.getDefault()
    java.util.TimeZone.setDefault(java.util.TimeZone.getTimeZone(zoneId))
    try {
        return block()
    } finally {
        java.util.TimeZone.setDefault(previous)
    }
}

Tests that would fail today only if the default became UTC (they specify the desired contract, they do not
reproduce a bug):

private val shiftedZones = listOf("UTC", "Europe/Berlin", "Pacific/Kiritimati", "America/Los_Angeles")

@Test
fun `converting a Long to a LocalDateTime does not depend on the default time zone`() {
    val millis = 1_704_110_400_123L // 2024-01-01T12:00:00.123Z
    shiftedZones.forEach { zone ->
        zone.asClue {
            withDefaultTimeZone(zone) {
                columnOf(millis).convertToLocalDateTime()[0] shouldBe
                    LocalDateTime(2024, 1, 1, 12, 0, 0, 123_000_000)
            }
        }
    }
}

@Test
fun `converting a LocalDate to an Instant does not depend on the default time zone`() {
    shiftedZones.forEach { zone ->
        zone.asClue {
            withDefaultTimeZone(zone) {
                columnOf(LocalDate(2024, 1, 1)).convertTo<Instant>()[0] shouldBe
                    Instant.parse("2024-01-01T00:00:00Z")
            }
        }
    }
}

A test that can be added right now, independent of which direction is chosen — it closes the hole left by
ParserTests and passes either way:

@Test
fun `the zone argument is honoured`() {
    val col = columnOf(1_704_110_400_123L) // 2024-01-01T12:00:00.123Z
    col.convertToLocalDateTime(TimeZone.UTC)[0] shouldBe LocalDateTime(2024, 1, 1, 12, 0, 0, 123_000_000)
    col.convertToLocalDateTime(TimeZone.of("Europe/Berlin"))[0] shouldBe
        LocalDateTime(2024, 1, 1, 13, 0, 0, 123_000_000)
}

Cases 3, 5 and 6 each deserve one test in the same shape.

Possible directions

Not mutually exclusive, and listed without a recommendation.

A. Resolve against UTC by default. Deterministic out of the box, and consistent with what ApacheArrow.md and
Parquet.md already promise. It is a behavioural breaking change for anyone relying on the ambient zone — though
no existing value-level test asserts the current behaviour, so the change is currently unobservable in CI, which is
an argument both for the change being safe and for it needing new tests either way.

B. Keep the system default, close the gaps. Add zone: TimeZone wherever it is missing (case 5: Instant → LocalDate/LocalTime, the whole local → absolute direction, Byte/Short, java.time.* sources), make it
reachable from an erased receiver (case 4), and document the default in convert.md and in the KDoc of the
zone-free overloads. Breaks nothing, but leaves the default non-deterministic.

C. Require an explicit zone. Deprecate the zone-less overloads for zone-dependent pairs. Most correct, most
disruptive.

B is a subset of the work needed under any of the three, since the gaps in case 5 leave users with no workaround
except mutating the JVM default.

Orthogonal to all three: convert.md should say what the current rule is. It says nothing today.

Notes

• The compiler plugin lives in the Kotlin repository
  (github.com/JetBrains/kotlin/tree/master/plugins/kotlin-dataframe), not in this one — this repo's
  plugins/kotlin-dataframe is a disabled, out-of-date legacy copy (see plugins/AGENTS.md). Adding or removing a
  parameter on any @Interpretable-annotated convert function therefore needs a matching interpreter change
  there; changing only the converter internals does not.
• Both api/convert.kt and impl/api/convert.kt are shaded into dataframe-compiler-plugin-core, which the
  compiler plugin (and IntelliJ) bundle to run compile-time interpreters — so a change here ships inside the
  plugin, and PluginApiUsages.kt in that module exercises convert directly.
• Related: Support Parquet UTC timestamp vectors (TimeStampXXXTZVector) in Arrow reader #926 (Arrow/Parquet timestamps with a time zone — the downstream half of this),
  Handle deprecation of kotlinx datetime Instant (and Clock) #1350 (kotlinx.datetime.Instant → kotlin.time.Instant migration, which duplicated many of these table
  entries).
• Minor, adjacent: the KDoc of the toLocalDateTime(zone) overloads says "interpret the … timestamp as a time"
  — a copy-paste from toLocalTime (api/convert.kt:2158, :2180, :2202, :2224).

[Jolanrensen]

Okay, this is a lot of text, but I agree in some parts: Instant -> LocalDate(Time) should indeed have an overload that allows a specified time zone.
That said, it's probably fine to keep using the system default timezone by default. This is the same logic we use for Locale after all. This does provide the user with an escape hatch, simply by calling java.util.TimeZone.setDefault().