how to fit an SLDS model with different observation models for each state

I was trying to fit an SLDS model with different observation models for each state, 
using 'slds = ssm.SLDS(emissions_dim, n_discrete_states, latent_dim, emissions="gaussian_orthog", single_subspace=False)'
but no optimization algorithm worked in this case, with error reported to be infinite value for variance of gaussian distribution.
