Friday, October 21, 2011

Why is the error function minimized in logistic regression convex?

We want to prove that the error/objective function of logistic regression :

is convex.

Proof:

Before beginning the proof, i would first like to make you review/recollect a few definitions/rules/facts/results related to convex functions:


  • Definition of a convex function: A function f(x) is said to be convex if the following inequality holds true:


  • First-order condition of convexity: A function f(x) which is differentiable is convex if the following inequality condition holds true:

Intuitively, this condition says that the tangent/first-order-taylor-series approximation of f(x) is globally an under-estimator.

  • Second-order condition of convexity: A function f(x) which is twice-differentiable is convex if and only if its hessian matrix (matrix of second-order partial derivatives) is positive semi-definite, i.e.

     
  • Sum/Linear-combination of two or more convex functions is also convex: Let f(x) and g(x) be two convex functions. Then any linear combination of these two functions



    is also a convex function (this can be easily proved using the definition of the convex function).

Now notice that if we can prove that the two functions



are convex, then our objective function



must also be convex since any linear combination of two or more convex functions is also convex.

Let us now try to prove that



is a convex function of theta. In order to do this, we will use the second-order condition of convexity described above. Let us first compute the hessian matrix:

Now below is the proof that this hessian matrix is positive semi-definite:

Let us now try to prove that



is a convex function of theta. In order to do this, we will again use the second-order condition of convexity described above. Let us first compute its hessian matrix:

Above, we have proved that both

are convex functions. And, the error/objective function of logistic regression

is essentially a linear-combination of several such convex functions. Now, since a linear combination of two or more convex functions is convex, we conclude that the objective function of logistic regression is convex.

Hence proved …

Following the same line of approach/argument it can be easily proven that the objective function of logistic regression is convex even if regularization is used.

15 comments:

  1. Just to correct a little mistake: only a positive linear combination of convex functions is guaranteed to be convex again. However, in the logistic regression case y_i are positive, so it works indeed.

    ReplyDelete
  2. This comment has been removed by the author.

    ReplyDelete
  3. I think for equation(2) you lost a z in the left part.

    ReplyDelete
  4. Awesome post, thank you.

    Agree with above that (2) missing a z on L.H.S.

    ReplyDelete
  5. I think in Eq.(2) you should write grad_{\theta} and not w.r.t. x.
    This is of course just a typo.
    Nice post btw.!

    ReplyDelete
  6. Thank you a lot! It has been very useful to me :)

    ReplyDelete
  7. who know paper proves that this loss function is convex

    ReplyDelete
  8. Hello Yorkies From Elvis Yorkshire Terrier - Specializing in Teacup Yorkies
    We are small breeders whose goal is to produce healthy, high quality little teacup size Yorkie puppies for sale. In order to have puppies for sale most of the time we have teamed up with a couple of other Yorkie breeders that have the same goals in mine. In fact we have some of their Yorkie breeders and they have some of our breeders.
    yorkies for sale, teacup yorkie puppies for sale, Yorkie puppies for sale,

    ReplyDelete
  9. f you are looking for a new furry friend, british shorthair kitten may be a perfect choice. british shorthair for sale near me have been around since the 1800s and were originally bred in England to hunt rats. These days they are more likely found as pets than on the job, but their hunting instincts still remain strong!  british shorthair cat for sale can come in many different colors so no matter what your preference is you will find a British Shorthair that matches it perfectly.

    ReplyDelete
  10. Very nice blog. Keep writing. Turkish evisa is a legalized entry Permit which is connected to the person’s passport that allows one to enter into Turkey for touristic purposes and many other purposes.


    ReplyDelete

  11. Hello everyone, Indian e visa application is an online travel authorization to travel to the country for tourism / business / medical / conference purposes. You can find out all the information about Indian visas through our website.

    ReplyDelete