A DDPM or DDIM samples images by denoising Gaussian noise, and the noise vector it starts from is, by construction, not a meaningful representation of anything — it's just where the reverse process happened to begin. GANs, by contrast, often end up with latent spaces (StyleGAN's \(\mathcal{W}\), for instance) where walking in a straight line changes one interpretable attribute at a time. For a course project in EE6180 (Topics in Advanced AI) at IIT Madras, we reproduced Diffusion Autoencoders (Preechakul, Chatthee, Wizadwongsa, and Suwajanakorn, CVPR 2022), which asks a direct question: can a diffusion model have that kind of latent space too, without giving up the generation quality that makes diffusion models worth using in the first place?
The idea: split the latent into semantic and stochastic parts
DiffAE's answer is to stop treating the diffusion model as a single, unconditioned generator and instead give it two separate sources of information about the image it's reconstructing:
- A semantic code \(z_{\text{sem}}\), produced by a learned encoder network that looks at the input image once and compresses it into a compact vector — conceptually the same job an ordinary autoencoder's bottleneck does, except this encoder's only job is to give the diffusion decoder something to condition on.
- A stochastic subcode \(x_T\), which is exactly the noise a DDIM would arrive at if you ran its deterministic forward (encoding) process on the same image. Because DDIM's forward and reverse processes are deterministic given a fixed noise schedule, \(x_T\) can be exactly recovered from the image and exactly inverted back to it — it's not free-floating noise, it's a stand-in for whatever fine detail the semantic code doesn't need to carry.
The decoder is a conditional DDIM: instead of only ever seeing a noisy image at each denoising step, it also sees \(z_{\text{sem}}\), so the semantic code steers what gets reconstructed while the stochastic subcode and the iterative denoising handle the high-frequency detail that would be wasteful to force through a low-dimensional bottleneck. That split is the whole trick: it gives you a compact, meaningful code for the parts of the image you'd actually want to edit, without asking that code to also memorize every pixel-level detail a generative model needs to look convincing.
Why "decodable" is the harder half of the claim
A compact code is easy to get — any autoencoder bottleneck gives you one. The paper's stronger claim is that \(z_{\text{sem}}\) is decodable in the sense that matters for editing: simple linear directions in that space correspond to interpretable attributes. To demonstrate this, the authors train a separate linear classifier on \(z_{\text{sem}}\) against CelebA-HQ attribute labels (smiling, age, and so on), then manipulate an image by moving its semantic code along the direction that classifier assigns to an attribute and decoding the result. If the space really is meaningfully organized, that's enough to produce a convincing "make this face smile more" edit without any attribute-specific retraining of the generator itself.
What we actually reproduced
Training DiffAE from scratch is not something you do on a laptop — the paper's FFHQ256 run needed 8×V100s, and even the smaller FFHQ128 configuration wants 4×V100s for the base model plus a separate pass for the latent DPM. Our reproduction used the authors' released pretrained checkpoints (FFHQ256, with the latent DPM used for sampling) and focused on the inference-side pipeline end to end, on our own photo rather than a dataset sample:
- Alignment. The model is trained on FFHQ-style aligned crops, so a raw photo has to be run through
align.py(dlib-based facial landmark alignment) before it means anything to the encoder. - Autoencoding. Encode the aligned photo to \((z_{\text{sem}}, x_T)\), then decode straight back, as the most basic check that the two-part latent actually reconstructs the input rather than a generic-looking face.
- Interpolation. Encode two images, interpolate both parts of their latents, and decode the intermediate points — a qualitative check that intermediate latents decode to intermediate, still-coherent faces rather than to noise.
- Manipulation. Shift \(z_{\text{sem}}\) along an attribute direction from the CelebA-HQ-trained linear classifier and decode, to see the semantic edit the paper's decodability claim promises.
The repository's own README table shows this end to end on one face — original photo, aligned crop, and the manipulated result, side by side — which is the same three-stage pipeline we ran.
The unglamorous part: making a 2022 codebase run in 2025
Reproduction work is disproportionately about environment friction rather than the method itself, and this was no exception. The original codebase pins against a numpy API surface that had moved on by the time we ran it, and getting experiment.py working again meant tracking down and fixing the specific calls that had changed behavior or been deprecated — not a conceptual obstacle, just the ordinary tax of running research code a few years after it was written. It's a useful reminder that "reproducing a paper" and "reproducing a paper's results" are different amounts of work: the second one is mostly done once you've gotten the first one to import cleanly.
What the exercise actually clarified
Running the pipeline ourselves made the semantic/stochastic split feel less like a modeling trick and more like an answer to a specific question: where should a generative model's capacity for exact detail live, if you also want a compact code you can reason about and edit? Forcing all of that into one bottleneck gets you either a code too high-dimensional to interpret or a decoder too constrained to look sharp. DiffAE's split — deterministic, invertible stochastic detail on one side, a compact and (by construction of the classifier experiment) linearly-decodable semantic code on the other — is a genuinely different way of allocating that capacity than treating the whole model as a black box you probe after the fact, which is the framing most interpretability work outside of generative modeling starts from. That contrast is what made this worth doing as a course project rather than just reading the paper.
This is a reproduction and exploration exercise, not new research — all credit for the method belongs to Preechakul, Chatthee, Wizadwongsa, and Suwajanakorn. Our fork (with the environment fixes and our own example images) is at github.com/aayushmanda/diffae; the official implementation and project page are at diff-ae.github.io. An LLM was used only for organizing this write-up from our own run notes and the paper; the reproduction itself was our own work for the course.
Reference
- Preechakul, K., Chatthee, N., Wizadwongsa, S., and Suwajanakorn, S. Diffusion Autoencoders: Toward a Meaningful and Decodable Representation. CVPR, 2022.