Implement gradient descent in python

Question

I am trying to implement gradient descent in python. Though my code is returning result by I think results I am getting are completely wrong.

Here is the code I have written:

import numpy as np
import pandas

dataset = pandas.read_csv('D:\ML Data\house-prices-advanced-regression-techniques\\train.csv')

X = np.empty((0, 1),int)
Y = np.empty((0, 1), int)

for i in range(dataset.shape[0]):
  X = np.append(X, dataset.at[i, 'LotArea'])
  Y = np.append(Y, dataset.at[i, 'SalePrice'])

X = np.c_[np.ones(len(X)), X]
Y = Y.reshape(len(Y), 1)

def gradient_descent(X, Y, theta, iterations=100, learningRate=0.000001):
  m = len(X)
  for i in range(iterations):
    prediction = np.dot(X, theta)
    theta = theta - (1/m) * learningRate * (X.T.dot(prediction - Y))

  return theta

  theta = np.random.randn(2,1)
  theta = gradient_descent(X, Y, theta)
  print('theta',theta)

The result I get after running this program is:

theta [[-5.23237458e+228] [-1.04560188e+233]]

Which are very high values. Can someone point out the mistake I have made in implementation.

Also, 2nd problem is I have to set value of learning rate very low (in this case i have set to 0.000001) to work other wise program throws an error.

Please help me in diagnosis the problem.

Code here : github.com/0xPrateek/ML-Algorithms/blob/master/Algorithms/… is self-explained although it's in component form implementation but will be helpful to understand — 0xPrateek
– 0xPrateek, Commented Jul 4, 2019 at 12:17
@0xPrateek RuntimeWarning: invalid value encountered in subtract — Bagira
– Bagira, Commented Jul 4, 2019 at 12:23
@0xPrateek The link you provided is very verbose. I am trying to implement gradient descent using numpy's ability to do vector calculations. That makes things a lot simpler. — Bagira
– Bagira, Commented Jul 4, 2019 at 12:29
In the line theta = theta - (1/m) * learningRate * (X.T.dot(prediction - Y)), theta is a 1d vector, but (1/m) * learningRate * (X.T.dot(prediction - Y)) is a single value. While the operation is valid in numpy, it is meaningless in mathematics. — Leporello
– Leporello, Commented Jul 4, 2019 at 12:49

Harsh Chaturvedi · Accepted Answer · 2019-07-04 13:33:46Z

1

try to reduce the learning rate with iteration otherwise it wont be able to reach the optimal lowest.try this

import numpy as np
import pandas

dataset = pandas.read_csv('start.csv')

X = np.empty((0, 1),int)
Y = np.empty((0, 1), int)

for i in range(dataset.shape[0]):
  X = np.append(X, dataset.at[i, 'R&D Spend'])
  Y = np.append(Y, dataset.at[i, 'Profit'])

X = np.c_[np.ones(len(X)), X]
Y = Y.reshape(len(Y), 1)

def gradient_descent(X, Y, theta, iterations=50, learningRate=0.01):
  m = len(X)
  for i in range(iterations):
    prediction = np.dot(X, theta)
    theta = theta - (1/m) * learningRate * (X.T.dot(prediction - Y))
    learningRate/=10;

  return theta

theta = np.random.randn(2,1)
theta = gradient_descent(X, Y, theta)
print('theta',theta)

answered Jul 4, 2019 at 13:33

Harsh Chaturvedi

7897 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Implement gradient descent in python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related