Differences

This shows you the differences between two versions of the page.

--- assignments:assignment3 [2016/09/15 14:43]
asa
+++ assignments:assignment3 [2016/09/15 14:44]
asa
@@ Line 1: / Line 1: @@
-~~NOTOC~~
+====== Assignment 2 ======
-====== Assignment 3 ======
+**Due:** 10/1 at 11:59pm.
-**Due:** 10/2 at 11pm.
-===== Preliminaries =====
 In this assignment you will explore ridge regression applied to the task of predicting wine quality.
 You will use the [[http://archive.ics.uci.edu/ml/datasets/Wine+Quality | wine quality]] dataset from the UCI machine learning repository, and compare accuracy obtained using ridge regression to the results from a [[http://www.sciencedirect.com/science/article/pii/S0167923609001377# | recent publication]] (if you have trouble accessing that version of the paper, here's a link to a [[http://www3.dsi.uminho.pt/pcortez/wine5.pdf| preprint]].
-The wine data is composed of two datasets - one for white wines, and one for reds.  In this assignment perform all your analyses on just the red wine data.
+The wine data is composed of two datasets - one for white wines, and one for reds.  Perform all your analyses on both.
-The features for the wine dataset are not standardized, so make sure you do this, especially since we are going to consider the magnitude of the weight vector (recall that standardization entails subtracting the mean and then dividing by the standard deviation for each feature; you can use the [[http://docs.scipy.org/doc/numpy/reference/routines.statistics.html | Numpy statistics module]] to perform the required calculations).
-==== Part 1 ====
-Implement ridge regression keeping the same API you used in implementing the classifiers in assignment 2, and functions for computing the following measures of error:
+=== Part 1 ===
+Implement ridge regression and functions for computing the following measures of error:
   * The Root Mean Square Error (RMSE).
@@ Line 28: / Line 22: @@
 $$MAD(h) = \frac{1}{N}\sum_{i=1}^N |y_i - h(\mathbf{x}_i)|.$$
-With the code you just implemented, your next task is to explore the dependence of error on the value of the regularization parameter, $\lambda$.
+Compute these measures of error for ridge regression applied
-In what follows set aside 30% of the data as a test-set, and compute the in-sample error, and the test-set error as a function of the parameter $\lambda$ on the red wine data.  Choose the values of $\lambda$ on a logarithmic scale with values 0.01, 0.1, 1, 10, 100, 1000 and plot the RMSE only.
+to the wine dataset over a range of the regularization parameter, $\lambda$ (choose values on a logarithmic scale, e.g. 0.01, 0.1, 1, 10, 100, 1000) and plot the results (use a fixed test set for computing them!)
-Repeat the same experiment where instead of using all the training data, choose 20 random training examples.
+The features for the wine dataset are not standardized, so make sure you do this, especially since we are going to consider the magnitude of the weight vector (recall that standardization entails subtracting the mean and then dividing by the standard deviation for each feature; you can use the [[http://docs.scipy.org/doc/numpy/reference/routines.statistics.html | Numpy statistics module]] to perform the required calculations).  What is the potential advantage of MAD over RMSE?
-Now answer the following:
+In addition to RMSE and MAD, plot the Regression Error Characteristic (REC) curve of a representative classifier.
+REC curves are described in the following [[http://machinelearning.wustl.edu/mlpapers/paper_files/icml2003_BiB03.pdf|paper]].
+What can you learn from this curve that you cannot learn from RMSE or MAD?
-  * What is the optimal value of $\lambda$?
+Compare the results that you are getting with the published results in the paper.
-  * What observations can you make on the basis of these plots?  (The concepts of overfitting/underfitting should be addressed in your answer).
-  * Finally, compare the results that you are getting with the published results in the paper linked above.  In particular, is the performance you have obtained is comparable to that observed in the paper?
-==== Part 2 ====
+=== Part 2 ===
-Regression Error Characteristic (REC) curves are an interesting way of visualizing regression error as described
-in the following [[http://machinelearning.wustl.edu/mlpapers/paper_files/icml2003_BiB03.pdf|paper]].
-Write a function that plots the REC curve of a regression method, and plot the REC curve of the best regressor you found in Part 1 of the assignment.
-What can you learn from this curve that you cannot learn from an error measure such as RMSE or MAD?
-==== Part 3 ====
 As we discussed in class, the magnitude of the weight vector can be interpreted as a measure of feature importance.
@@ Line 58: / Line 44: @@
 Next, perform the following experiment:
 Incrementally remove the feature with the lowest absolute value of the weight vector and retrain the ridge regression classifier.
-Plot RMSE as a function of the number of features that remain on the test set which you have set aside.
+Plot RMSE and MAD as a function of the number of features that remain on the test set which you have set aside.
 ===== Submission =====
@@ Line 91: / Line 77: @@
               Grammar, spelling, and punctuation.
 </code>

CS545 fall 2016

User Tools

Site Tools

Differences

Page Tools