Categorical Reparameterization with Denoising Diffusion models
Image generated by Gemini AI
A new paper introduces a diffusion-based soft reparameterization for optimizing categorical variables, enhancing existing continuous relaxations. This method utilizes a Gaussian noising process with an efficient closed-form denoiser, allowing for backpropagation without prior training. Experiments indicate that this approach offers competitive or improved performance on various benchmarks, addressing the challenges of noise and bias in traditional optimization methods.
Categorical Reparameterization Enhanced by Denoising Diffusion Models
A new study has introduced a diffusion-based soft reparameterization technique for optimizing categorical variables. This approach addresses the limitations of traditional score-function estimators and continuous relaxations used in optimization.
Standard methods often involve score-function estimators that are unbiased but have high noise levels. Continuous relaxations replace discrete distributions with smooth surrogates, allowing for pathwise gradients but optimizing biased, temperature-dependent objectives.
The authors propose a novel strategy leveraging a denoising diffusion process, providing a closed-form solution for the denoiser under a Gaussian noising process. This creates a training-free diffusion sampler that permits backpropagation, enhancing optimization.
The proposed method demonstrated competitive or superior performance across various benchmarks, indicating a significant advancement in optimizing categorical distributions.
Related Topics:
📰 Original Source: https://arxiv.org/abs/2601.00781v1
All rights and credit belong to the original publisher.