Least Squares Regression
This is one of the favorite topics for people to start machine learning. It has roots more than a century. The basic problem of finding the rule behind a process lead people to think about how to create a rule from scientific measurements. Offcourse, linear relationships are the most intuitive ones. If you buy one apple it is 1$ if you buy 10 it is 10$. The fomulation in this case would be y=wx for single variable. Hoever, machine learning is reverse of the process we are not trying to find y or total amount in previous example instead we would like to find w from data. Lets extend on this apples example. If we have data like this 10 apples -> 9$ 8 apples -> 8$ 2 apples -> 3$ What would be your guess of w (unit price) if you assume linear relationship ? Least square estimate is a learning mechanism by optimizing squared discripancies between model and data. So lets formulate like this argmin_w \sum (wx-y_true)^2 In most minimization problems you tak...