We want to prove that the error/objective function of logistic regression :
is convex.
Proof:
Before beginning the proof, i would first like to make you review/recollect a few definitions/rules/facts/results related to convex functions:
- Definition of a convex function: A function f(x) is said to be convex if the following inequality holds true:
- First-order condition of convexity: A function f(x) which is differentiable is convex if the following inequality condition holds true:
Intuitively, this condition says that the tangent/first-order-taylor-series approximation of f(x) is globally an under-estimator.
- Second-order condition of convexity: A function f(x) which is twice-differentiable is convex if and only if its hessian matrix (matrix of second-order partial derivatives) is positive semi-definite, i.e.
- Sum/Linear-combination of two or more convex functions is also convex: Let f(x) and g(x) be two convex functions. Then any linear combination of these two functions
is also a convex function (this can be easily proved using the definition of the convex function).
Now notice that if we can prove that the two functions
are convex, then our objective function
must also be convex since any linear combination of two or more convex functions is also convex.
Let us now try to prove that
is a convex function of theta. In order to do this, we will use the second-order condition of convexity described above. Let us first compute the hessian matrix:
Now below is the proof that this hessian matrix is positive semi-definite:
Let us now try to prove that
is a convex function of theta. In order to do this, we will again use the second-order condition of convexity described above. Let us first compute its hessian matrix:
Above, we have proved that both
are convex functions. And, the error/objective function of logistic regression
is essentially a linear-combination of several such convex functions. Now, since a linear combination of two or more convex functions is convex, we conclude that the objective function of logistic regression is convex.
Hence proved …
Following the same line of approach/argument it can be easily proven that the objective function of logistic regression is convex even if regularization is used.
Just to correct a little mistake: only a positive linear combination of convex functions is guaranteed to be convex again. However, in the logistic regression case y_i are positive, so it works indeed.
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteI think for equation(2) you lost a z in the left part.
ReplyDeleteAwesome post, thank you.
ReplyDeleteAgree with above that (2) missing a z on L.H.S.
I think in Eq.(2) you should write grad_{\theta} and not w.r.t. x.
ReplyDeleteThis is of course just a typo.
Nice post btw.!
Thank you a lot! It has been very useful to me :)
ReplyDeletewho know paper proves that this loss function is convex
ReplyDeleteشركة تعقيم و تطهير بحي العمامرة
ReplyDeleteشركة تعقيم بحي الياسمين
تعقيم و تطهير بحي الفجيرة
Hello Yorkies From Elvis Yorkshire Terrier - Specializing in Teacup Yorkies
ReplyDeleteWe are small breeders whose goal is to produce healthy, high quality little teacup size Yorkie puppies for sale. In order to have puppies for sale most of the time we have teamed up with a couple of other Yorkie breeders that have the same goals in mine. In fact we have some of their Yorkie breeders and they have some of our breeders.
yorkies for sale, teacup yorkie puppies for sale, Yorkie puppies for sale,
f you are looking for a new furry friend, british shorthair kitten may be a perfect choice. british shorthair for sale near me have been around since the 1800s and were originally bred in England to hunt rats. These days they are more likely found as pets than on the job, but their hunting instincts still remain strong! british shorthair cat for sale can come in many different colors so no matter what your preference is you will find a British Shorthair that matches it perfectly.
ReplyDeleteHelium miner for sale
ReplyDeleteRak hotspot miner V2 for sale
Cute Doberman puppies for sale
ReplyDeleteDoberman puppies near me
Very nice blog. Keep writing. Turkish evisa is a legalized entry Permit which is connected to the person’s passport that allows one to enter into Turkey for touristic purposes and many other purposes.
ReplyDelete
ReplyDeleteHello everyone, Indian e visa application is an online travel authorization to travel to the country for tourism / business / medical / conference purposes. You can find out all the information about Indian visas through our website.
ümraniye samsung klima servisi
ReplyDeletekartal mitsubishi klima servisi
ümraniye mitsubishi klima servisi
beykoz vestel klima servisi
üsküdar vestel klima servisi
ümraniye alarko carrier klima servisi
tuzla beko klima servisi
ataşehir lg klima servisi
çekmeköy alarko carrier klima servisi