[SOLVED] MachineLearning Homework 8-Anomaly Detection

Goal

● Unsupervised anomaly detection in computer vision: Whether a machine learning model is able to tell a testing image is of the same class (distribution) as the training images

Goal

● Unsupervised anomaly detection in computer vision: Whether a machine learning model is able to tell a testing image is of the same class (distribution) as the training images

Training

Model

Goal

● Unsupervised anomaly detection in computer vision: Whether a machine learning model is able to tell a testing image is of the same class (distribution) as the training images

Training

Testing

Model

Seen Normal

Model

Goal

● Unsupervised anomaly detection in computer vision: Whether a machine learning model is able to tell a testing image is of the same class (distribution) as the training images

Training

Testing

Model

Seen Normal

Model

Model

Unseen Anomaly

Data

• ●  Trainingset: About 140k human faces (size 64*64*3)
• ●  Testingset: Another 10k data from the same distributions as the

  trainingset (normal data, of class label 0) along with 10k human face

  images from the other distributions (anomalies, of class label 1)

• ●  Notice: Additional training data and pretrained models are prohibited
• ●  Data format: tar zxvf data-bin.tar.gz
• ●  data-bin/

  ○ trainingset.npy ○ testingset.npy

Method – Autoencoder

Autoencoder

• ●  When to stop training? Training should stop when the mse loss converges
• ●  During inference, we calculate the reconstruction error between the input

  image and the reconstructed one

• ●  The reconstruction error will be referred to as abnormality (anomaly

  score)

• ●  The abnormality of an image can be a metric of the possibility that it’s

  distribution is unseen during training

• ●  Therefore we use the abnormality as our predicted values

Accuracy score

• ●  Usually we compute accuracy scores for classification tasks
• ●  Here, our model functions as a sensor (or a detector) rather than a

  classifier

• ●  Thus, we need a threshold with respect to abnormality (usually the

  reconstruction error) to determine whether a piece of data is an anomaly

• ●  If we used accuracy score for this assignment, you would have to try every

  possible threshold for one single model to get a satisfactory score

• ●  However, what we want is a sensor that gets the highest accuracy on the

  average of every possible threshold

Which sensor is better?

Metric – ROC_AUC score

• ●  A good sensor should
  • ○  Give high anomaly scores to the anomalies and low scores to the normal data
  • ○  Exhibit a large gap between the scores of 2 groups
• ●  An ROC is suitable for our task
• ●  Each point on the ROC curve stands for true positive rate and false

  positive rate at certain threshold

• ●  The Area Under the ROC curve is calculated to measure the general ability

  of the model

ROC_AUC score

https://en.wikipedia.org/wiki/Receiver_operating_characteristic

ROC_AUC score

https://en.wikipedia.org/wiki/Receiver_operating_characteristic

Kaggle

Metric: ROC_AUC score Sample output:

How ROC AUC is calculated

Sort by score

https://towardsdatascience.com/how-to-calculate-use-the-auc-score-1fc85c9a8430

How ROC AUC is calculated

How ROC AUC is calculated

Area Under Curve: 0.5*1⁄3 + 2⁄3 = 0.8333

Scoring

• ●  Code submission: 4 pt
• ●  Baselines 6 pt (3 pt for the public ones and the other 3 pt for the private

ones)

• ○  Simple public: 1 pt
• ○  Medium public: 1 pt
• ○  Strong public:
• ○  Boss public:
• ○  Simple private:
• ○  Medium private: 1 pt
• ○  Strong private: 0.5 pt
• ○  Boss private: 0.5 pt

(public score: 0.64046) (public score: 0.75719) (public score: 0.81304) (public score: 0.86590)

● Bonus for submitting report: 0.5 pt

0.5 pt 0.5 pt

1 pt

Bonus

• ●  If you succeed in beating both boss baselines, you can get extra 0.5 pt by submitting a brief report to explain your methods (in less than 100 English words), which will be made public to the whole class
• ●  Report Template

Baseline guides

● Simple
○ FCN autoencoder

● Medium

• ○  CNN autoencoder
• ○  Try smaller models (less layers)
• ○  Smaller batch size

  ● Strong

• ○  Add BatchNorm
• ○  Train for longer

  ● Boss:

• ○  Add an extra classifier
• ○  Sample random noises as anomaly images
• ○  Or one-class-classification (OCC) with GANs: OCGAN, End-to-end OCC, paper pool for

Anomaly Detection

Baseline training statistics

● Simple

• ○  Number of parameters: 3176419
• ○  Training time on colab: ~ 30 min

  ● Medium

• ○  Number of parameters: 47355
• ○  Training time on colab: ~ 30 min

  ● Strong

• ○  Number of parameters: 47595
• ○  Training time on colab: 4 ~ 5 hrs

  ● Boss:

• ○  Number of parameters: 4364140
• ○  Training time on colab: 1.5~3 hrs

Strong baseline training curve

Loss

Steps

Code Submission

• ●  Zip your code and name the compressed file <student_id>_hw8.zip
• ●  And it should contain
  • ○  Your codes
  • ○  report.pdf (only for those beating both boss baselines)
• ●  Submit <student_id>_hw8.zip via NTU COOL

Code Submission

● DO

• ○  Specify the source of your code. (You may refer to Academic Ethics Guidelines)
• ○  Organize your code and make it easy to read (not necessary)

● DO NOT

• ○  Submit empty or garbage files
• ○  Submit the dataset or model
• ○  Compress your codes into other formats like .rar or .7z and simply rename it to .zip

  ● Note

• ○  We can only see your last submission
• ○  Do not submit your model or dataset
• ○  If your code is not reasonable, your semester grade x 0.9

Regulations

• ●  Plagiarism is not allowed
• ●  Do not modify your prediction file
• ●  Do not share your prediction file with anyone
• ●  Do not submit your prediction file more than 5 times to Kaggle in any way
• ●  Do NOT search or use additional data or pre-trained models.
• ●  Violators are subject to x 0.9 of their semester grades
• ●  Prof. Lee & TAs preserve the rights to change the rules & grades

I