Hyperspectral data acquired through remote sensing are invaluable for environmental and resource studies. While rich in spectral information, various complexities, such as environmental conditions, material properties, and sensor characteristics can cause significant variability even among pixels belonging to the same material class. This variability poses nuisance for accurate land-cover classification and analysis. Focusing on the spectral domain, we utilize an autoencoder architecture called the symmetric autoencoder (SymAE), which leverages permutation invariant representation and stochastic regularization in tandem to disentangle class-invariant “coherent” features from variability-causing “nuisance” features on a pixel-by-pixel basis. This disentanglement is achieved through a purely data-driven process, without the need for hand-crafted modeling, noise distribution priors, or reference “clean signals.” In addition, SymAE can generate virtual spectra through manipulations in latent space. Using AVIRIS instrument data, we demonstrate these virtual spectra offer insights on the disentanglement. Extensive experiments across six benchmark hyperspectral datasets show that coherent features extracted by SymAE can be used to achieve state-of-the-art pixel-based classification. Furthermore, we leverage these coherent features to enhance the performance of some leading spectral–spatial hyperspectral image (HSI) classification methods. Our approach especially shows improvement in scenarios where training and test regions are disjoint, a common challenge in real-world applications where existing methods often struggle to maintain relatively high performance.