Comments on: double descent
https://xianblog.wordpress.com/2019/11/07/double-descent/
an attempt at bloggin, nothing more...Thu, 07 Nov 2019 08:49:11 +0000
hourly
1 http://wordpress.com/
By: samuelwxy
https://xianblog.wordpress.com/2019/11/07/double-descent/comment-page-1/#comment-201157
Thu, 07 Nov 2019 08:49:11 +0000http://xianblog.wordpress.com/?p=40420#comment-201157I’ve also experienced similar phenomenon before when doing linear regression (or ridge regression with a small tuning parameter). The worst result is always achieved when n=p. My understanding is that the design matrix (X’X) has the worst condition number when n=p. From random matrix theory perspective, the singular value of the design matrix is lower bounded by max(p-n, n-p), which arrives at 0 when n=p. A bad condition number implies high variance in optimization and worse result. The dimension difference (regardless of large p or n) can actually induce a better condition number for the optimization problem.
]]>