Nonlinear Dimensionality Reduction with UMAP
This course project for MA4270 Data Modelling and Computation was my first systematic study of manifold learning and nonlinear dimensionality reduction.
UMAP and parametric UMAP
I focused on UMAP and parametric UMAP. In addition to the papers, I studied the reference implementation and documentation, then wrote a PyTorch implementation for learning purposes. The report develops the attractive and repulsive forces in the embedding objective and makes explicit how their balance changes with the fuzzy-graph weight $p_{ij}$.
Estimating intrinsic dimension
I also used probabilistic PCA on local neighborhoods to approximate tangent spaces and estimate intrinsic dimension. The underlying observation is that an $n$-dimensional smooth manifold has an $n$-dimensional tangent space. For a chart $\phi:U_\alpha\to\mathbb{R}^n$, the differential
\[d\phi_p:T_pM\longrightarrow T_{\phi(p)}\mathbb{R}^n\]is a linear isomorphism. The following diagram expresses the equivalent construction through derivations on germs of smooth functions:
Consequently,
\[T_{\phi(p)}(\mathbb{R}^n) = \text{span}\left(\left{\frac{\partial}{\partial x_1}, \ldots, \frac{\partial}{\partial x_n}\right}\right)\]
has dimension $n$. Locally fitting a low-rank probabilistic PCA model is therefore a practical way to estimate the dimension of a sampled manifold, provided that the neighborhood is small enough to be approximately linear and large enough to overcome noise.
Reflection
This topic interested me because non-Euclidean configuration spaces arise naturally in robotics. I also wondered whether a force-directed embedding could warp inputs into a geometry better suited to stationary Gaussian-process kernels. An axis-aligned transformation such as a Beta-CDF input warp cannot represent general geometric distortion, but applying UMAP directly would make out-of-sample prediction and uncertainty propagation difficult. Deep kernel learning is probably the more direct formulation of that idea. The implementation in this project is educational rather than production-oriented; for practical use, I recommend the official UMAP package.