1. Logistic regression
Input:
1. (number of data points)
2.
Function:
- Generate data point:
independently sampled from
- Generate data point: independently sampled from
- Use Logistic regression to separate
and steepest gradient descent method during optimization.In other words, when the Hessian is singular, use steepest descent for instead. You should come up with a reasonable rule to determine convergence.(a simple run out of the loop should be used as the ultimatum)
Output:
- The confusion matrix and the sensitivity and specificity of the logistic regression
applied to the training data .
- Visualization
Plot the ground truth Plot the predict result
Gradient descent Newton’s method
Use the Gaussian random number generator in homework 3.
Sample input & output (for reference only) Case 1:
( : mean,
and
and
: variance)
, where and are respectively.
, where and are respectively.
and . You should implement both Newton’s
|
1 2 3 |
Gradient descent: w: |
|
4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
-78.1766393662 6.7233419236 11.2430677919 Confusion Matrix:
Predict cluster 1 Predict cluster 2
Is cluster 1 50 0 Is cluster 2 0 50 Sensitivity (Successfully predict cluster 1): 1.00000 Specificity (Successfully predict cluster 2): 1.00000 ---------------------------------------- Newton's method: w: -118.3601516394 8.7747332848 10.1954120077 Confusion Matrix:
Predict cluster 1 Predict cluster 2
Is cluster 1 50 0 Is cluster 2 0 50 Sensitivity (Successfully predict cluster 1): 1.00000 Specificity (Successfully predict cluster 2): 1.00000 |
Case 2:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
Gradient descent: w: -71.1902536008 46.0123814025 54.6803199701 Confusion Matrix:
Predict cluster 1 Predict cluster 2
Is cluster 1 16 34 Is cluster 2 3 47 Sensitivity (Successfully predict cluster 1): 0.32000 Specificity (Successfully predict cluster 2): 0.94000 ---------------------------------------- Newton's method: w: -1.9045831451 0.3940876974 0.5695243849 Confusion Matrix:
Predict cluster 1 Predict cluster 2
Is cluster 1 40 10 Is cluster 2 10 40 Sensitivity (Successfully predict cluster 1): 0.80000 Specificity (Successfully predict cluster 2): 0.80000 |
2. EM algorithm
Input: MNIST training data and label sets. (Same as HW02) Function:
- Binning the gray level value into two bins. Treating all pixels as random variables following Bernoulli distributions. Note that each pixel follows a different Binomial distribution independent to others.
- Use EM algorithm to cluster each image into ten groups. You should come up with a reasonable rule to determine convergence. (a simple run out of the loop should be used as the ultimatum)
Output:
- For each digit, output a confusion matrix and the sensitivity and specificity of the clustering applied to the training data.
- Print out the imagination of numbers in your classifier
Just like before, about the details please refer to HW02
Hint: The algorithm is a kind of unsupervised learning, so the labels are not used during training. But you can use these labels to help you to figure out which class belongs to which number.
In other words, you should find a way to assign label to each class which you classified
before you compute the confusion matrix Sample input & output (for reference only)
|
1 2 3 |
class 0: 0000000000000000000000000000 0000000000000000000000000000 |
|
4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000100000000000 0000000000000001110000000000 0000000000000011110000000000 0000000000000011100000000000 0000000000000011000000000000 0000000000000001000000000000 0000000000000000000000000000 0000000000000010000000000000 0000000000000110000000000000 0000000000000110000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 class 1: 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000001100000000000000 0000000000000000111000000000 0000000000000000111000000000 0000000000000000111000000000 0000000000000000011000000000 0000000000000000011000000000 0000000000000000111000000000 0000000000000000111000000000 0000000000000001110000000000 0000000000000111111000000000 0000000000001111111000000000 0000000000001111111000000000 0000000000011100011000000000 0000000000000000000000000000 |
53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 |
0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 ... all other unlabeled imagination of numbers goes here ... class 9: 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000011110000000000 0000000000001111111000000000 0000000000011111111100000000 0000000000000000011100000000 0000000000000000001000000000 0000000000000000001000000000 0000000000000011111000000000 0000000000000111111100000000 0000000000011111111100000000 0000000000000000001100000000 0000000000000000001100000000 0000000001000000000100000000 0000000010000000000100000000 0000000010000000001100000000 0000000110000000011100000000 0000000110000000111000000000 0000000111000100110000000000 0000000011111011000000000000 0000000001111110000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 No. of Iteration: 1, Difference: 3176.579389514846 ------------------------------------------------------------ class 0: 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 |
102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 |
0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000001100000000000 0000000000000011100000000000 0000000000000011100000000000 0000000000000111000000000000 0000000000000111000000000000 0000000000000111000000000000 0000000000000111000000000000 0000000000000010000000000000 0000000000000010000000000000 0000000000000110000000000000 0000000000000010000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 ... all other iterations goes here ... class 9: 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000001110000000000 0000000000000011110000000000 0000000000000011110000000000 0000000000000111000000000000 0000000000000110000000000000 0000000000001100000000000000 0000000000001100000000000000 0000000000011100000000000000 0000000000011100000000000000 0000000000011000100000000000 0000000000111001111000000000 0000000000110000111000000000 0000000000110000011000000000 0000000000110000011000000000 0000000000110000111000000000 0000000000110001111000000000 0000000000111111110000000000 |
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 |
0000000000111111100000000000 0000000000011110000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 No. of Iteration: 10, Difference: 19.89546432548733 ------------------------------------------------------------ ------------------------------------------------------------ labeled class 0: 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000111110000000000 0000000000011111111000000000 0000000000111111111110000000 0000000001111111111110000000 0000000011111000000111000000 0000000011110000000011100000 0000000111100000000011100000 0000000111000000000001100000 0000001111000000000001110000 0000001110000000000001110000 0000001110000000000001110000 0000001110000000000011100000 0000001110000000000011100000 0000001110000000000111000000 0000001111000000001111000000 0000000111100000111110000000 0000000011111111111100000000 0000000001111111110000000000 0000000000011111000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 labeled class 1: 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 |
200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 |
0000000000000000000000000000 0000000000000000000000000000 0000000000000001100000000000 0000000000000011100000000000 0000000000000011100000000000 0000000000000011100000000000 0000000000000011100000000000 0000000000000011000000000000 0000000000000111000000000000 0000000000000111000000000000 0000000000000111000000000000 0000000000000111000000000000 0000000000001110000000000000 0000000000001110000000000000 0000000000001110000000000000 0000000000001110000000000000 0000000000001100000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 ... all other labeled imagination of numbers goes here ... labeled class 9: 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000111000000000000000 0000000001110000011000000000 0000000011100000011000000000 0000000011000000011000000000 0000000011000000011000000000 0000000011000000111000000000 0000000011000001111000000000 0000000011111111111000000000 0000000001111111111000000000 0000000000000000111000000000 0000000000000000110000000000 0000000000000000110000000000 0000000000000000110000000000 0000000000000000110000000000 |
249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 |
0000000000000000100000000000 0000000000000000100000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 0000000000000000000000000000 ------------------------------------------------------------ Confusion Matrix 0:
Predict number 0 Predict not number 0
Is number 0 3023 2900 Isn't number 0 113 53964 Sensitivity (Successfully predict number 0) : 0.51038 Specificity (Successfully predict not number 0): 0.99791 ------------------------------------------------------------ Confusion Matrix 1:
Predict number 1 Predict not number 1
Is number 1 5986 756 Isn't number 1 800 52458 Sensitivity (Successfully predict number 1) : 0.88787 Specificity (Successfully predict not number 1): 0.98498 ------------------------------------------------------------ ... all other confusion matrix goes here ... ------------------------------------------------------------ Confusion Matrix 9:
Predict number 9 Predict not number 9
Is number 9 2718 3231 Isn't number 9 5147 48904 Sensitivity (Successfully predict number 9) : 0.45688 Specificity (Successfully predict not number 9): 0.90478 Total iteration to converge: 10 Total error rate: 0.5081666666666667 |
