Extend niche analysis methods beyond `nhood_kmeans` and `nhood_lda`

[ArneDefauw]

Harpy currently supports niche analysis via:

• nhood_kmeans: hard clustering on neighborhood composition
• nhood_lda: soft topic modeling on neighborhood cell-type counts

These are useful starting points, but niche analysis can be approached in several additional ways depending on whether the goal is hard assignment, soft assignment, compositional interpretation, multiscale structure, or stronger use of the spatial graph itself.

This issue proposes expanding Harpy’s niche-analysis toolbox with additional methods that build on the same neighborhood graph / neighborhood composition framework already present in harpy.table.niches.

Why this is useful

Different biological questions are better served by different niche models:

• some users want hard niche labels
• some want soft mixtures or overlapping niches
• some want methods tailored to compositional/count data
• some want methods that more directly exploit the spatial graph
• some want multiscale or hierarchical niche structure

Providing a broader set of methods would make Harpy more flexible while keeping a unified API around neighborhood-based niche analysis.

Proposed methods

1. NMF-based niche analysis

   • Apply Non-negative Matrix Factorization to neighborhood count matrices.
   • Input: nhood_counts
   • Output:
     • per-cell component weights in adata.obsm[...]
     • component-by-cell-type profiles in adata.uns[...]
     • optional hard niche label via argmax
   • Motivation:
     • similar interpretability to LDA
     • additive, non-negative components
     • often easier to explain and use than probabilistic topic models

2. Graph community detection on niche similarity

   • Build a graph of neighborhood similarity between cells, then run Leiden or Louvain.
   • Possible variants:
     • use the existing spatial graph with reweighted edges
     • build a new graph from neighborhood-composition similarity
   • Output:
     • hard niche labels in adata.obs[...]
   • Motivation:
     • makes stronger use of graph structure
     • potentially captures communities that centroid-based clustering misses

3. Gaussian mixture models (GMM) on neighborhood profiles

   • Fit a Gaussian mixture model on neighborhood fractions or a low-dimensional embedding thereof.
   • Output:
     • soft assignment matrix in adata.obsm[...]
     • hard niche labels in adata.obs[...]
   • Motivation:
     • soft clustering alternative to KMeans
     • simpler probabilistic model than LDA

4. PCA/SVD + clustering baseline

   • Apply PCA or truncated SVD on neighborhood features, then cluster in low-dimensional space.
   • Candidate clustering backends:
     • KMeans
     • GMM
     • Leiden on a kNN graph in latent space
   • Motivation:
     • strong and simple baseline
     • useful for large datasets and benchmarking

5. Hierarchical clustering of neighborhood composition

   • Cluster cells or aggregated niche centroids using hierarchical clustering.
   • Output:
     • dendrogram-compatible structure
     • optional cuts at different niche resolutions
   • Motivation:
     • useful for multiscale or nested niche organization
     • interpretable relationships between niche states

6. Compositional-data-aware niche analysis

   • Treat normalized neighborhood fractions explicitly as compositional data.
   • Possible workflow:
     • CLR or ILR transform
     • clustering on transformed values
   • Possible distance metrics:
     • Aitchison
     • Jensen-Shannon
     • cosine
   • Motivation:
     • neighborhood fractions are compositional by nature
     • may improve behavior over plain Euclidean clustering

7. Spatially smoothed niche models

   • Add explicit spatial regularization or smoothing to niche assignments.
   • Possible methods:
     • Hidden Markov random field
     • Potts-model-like smoothing
     • graph-based label smoothing after initial clustering
   • Motivation:
     • niche assignments should often vary smoothly in tissue
     • may reduce noisy cell-level label switching

8. Graph neural network / graph autoencoder niche embeddings

   • Learn embeddings from both cell identity and graph structure before clustering.
   • Output:
     • niche embedding in obsm
     • hard/soft niche labels downstream
   • Motivation:
     • richer use of topology and local context
     • potentially better scalability to more complex graphs

9. Spatial topic models

   • Extend topic modeling so topic proportions are spatially structured or smoothed.
   • Motivation:
     • LDA currently ignores spatial continuity beyond the neighborhood counts
     • spatially aware topic models may better capture tissue domains

Suggested implementation order

Recommended order of implementation based on impact vs complexity:

1. nhood_nmf
2. graph/community-based niche clustering
3. nhood_gmm
4. compositional-data workflows
5. hierarchical / multiscale methods
6. more advanced spatial or neural models

Suggested API direction

New methods should ideally follow the current nhood_kmeans / nhood_lda pattern:

• reuse precomputed spatial connectivities
• reuse or derive nhood_counts / nhood_composition
• write hard labels to adata.obs[key_added] when applicable
• write soft assignments / latent representations to adata.obsm[...]
• write model metadata and component/topic/niche profiles to adata.uns[...]

Examples:

• nhood_nmf(...)
• nhood_gmm(...)
• nhood_leiden(...) or nhood_graph_clustering(...)
• nhood_hierarchical(...)

Design considerations

• Decide which methods operate on:
  • nhood_counts
  • nhood_composition
  • transformed or latent embeddings
• Keep isolated-cell behavior consistent across methods
• Keep metadata conventions consistent:
  • keys in .obs, .obsm, .uns
• Add small deterministic unit tests similar to the current niche tests
• Prefer interpretable outputs that map back to cell-type categories