fix(spark): use .svc short form for Spark Connect URL (fixes #490)

[adiprathapa]

build_service_url() in kubeflow/spark/backends/kubernetes/utils.py emits Spark Connect URLs used inside the cluster with the FQDN suffix .svc.cluster.local, producing URLs like sc://my-session-svc.spark.svc.cluster.local:15002. The cluster DNS short form .svc resolves to the same Service in any standard cluster and matches the format used in KEP-107 examples (sc://spark-cluster.spark-system.svc:15002). This switches the format to sc://{service}.{namespace}.svc:{port} and aligns the unit and backend tests. Two e2e docstrings (test/e2e/spark/run_in_cluster.py, test/e2e/spark/test_spark_examples.py) that referenced the old hostname are also refreshed.

All 101 spark tests and 389 tests in the full python suite pass. ruff check and ruff format are clean on the changed files.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign electronic-waste for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
• Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

• Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Shortens the in-cluster Spark Connect service URL from the fully qualified *.svc.cluster.local form to *.svc, relying on DNS search domains to resolve the suffix.

Changes:

• Update build_service_url to emit sc://<service>.<namespace>.svc:<port> instead of the fully qualified .svc.cluster.local form.
• Update related unit tests and e2e test assertions/comments to match the new URL form.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
kubeflow/spark/backends/kubernetes/utils.py	Drop .cluster.local suffix from generated Spark Connect URL.
kubeflow/spark/backends/kubernetes/utils_test.py	Update expected URL in unit test.
kubeflow/spark/backends/kubernetes/backend_test.py	Adjust expected substring in backend test.
test/e2e/spark/run_in_cluster.py	Update docstring to reflect new URL form.
test/e2e/spark/test_spark_examples.py	Update comment to reflect new URL form.