Abstract: In order to obtain an optimal tradeoff between cross-term reduction and high time-frequency concentration in time-frequency distribution of Cohen's class, an optimization algorithm of kernel parameters based on third-order Rényi entropy was proposed. From the asymptotic cross term invariance of third-order Rényi entropy, the optimal kernel parameters can be obtained by searching the transition of the curve of third-order Rényi entropy versus kernel parameters. The theoretical analysis and simulation results show that the optimization of kernel parameters based on third-order Rényi entropy can match the kernel function best with signals to yield a high-performance time-frequency distribution.