[SOLVED] NASA Project-Near-Earth Objects

Agenda

• –  About the dataset
• –  Exploratory Analysis
• –  Solutions and Insights
• –  Conclusion and next steps

Problem Statement:

Can we predict through certain variables whether a certified asteroid will be hazardous or not?

Warning!!!

About the Dataset

• –  90836 rows
• –  10 columns
• –  6 variables

Variables Analysis & Cleaning

• ●  Estimated diameter min and estimated diameter max are correlated so one of them will be removed
• ●  Orbiting body and sentry object will be removed
• ●  Columns like name, id will also be removed

Correlation between the Variables of interest

Absolute Magnitude

A measure of luminescence used for measuring the approximate diameter of an asteroid.

Estimated Diameter

• –  About 75% of all hazardous are from the range of .25 to 1 kilometer. While only accounting for 1⁄6 of the total number of asteroids
• –  Asteroids smaller than .25 kilometers were only hazardous < 1% of the time while those larger were hazardous 40% of the time.

Naive Bayes

Qcut: Estimated Diameters

High amount of data are relatively small, but the range is large! Results in importance actually fits our estimation.

Naive Bayes

Accuracy:

1. Prior Probability for No Hazard: 90.15%
2. Naive Bayes:
   1. Training data: 89.77%
   2. Testing data: 89.46%

Likelihoods→Importance:

F1 Score:

1. Precision score: 27.9% 2. Recall score: 3.6%

3. F1 Score

1. Training data: 6.0%
2. Testing data: 6.4%

Logistic Regression

Accuracy: -Baseline: -Prediction:

Variable Importance:

• –  Most significant: –

  –

• –  Least significant:
  – relative_velocity

0.9024298183533852

0.9032696047851455

est_diameter_min

est_diameter_max

K- Nearest Neighbors

Baseline:

0.9024298183533852

Prediction:

0.9031228211808741

F-Score = .0006

Trees and Ensemble Methods

• First we a tried to fit decision trees and other ensemble classifiers like Bagging, Random Forest and Gradient Boosting without playing too much with the parameters.
• The metrics obtained via these models are shown on the right.
• Clearly, the big decision Tree was overfitting with a training accuracy of 100% and a test accuracy of 89.4% (even lesser than the baseline accuracy).
• Even Bagging and Random Forest were overfitting with much higher accuracy values for the training datasets compared to the test datasets.
• Boosting seemed to be performing the best on the Test Set.

Trees and Ensemble Methods

• Next, we will try to tune the parameters for Gradient Boosting.
• On varying the no. of estimators across 100,200,300,400 and 500, we obtained the plot on the right. The differences are minuscule. We will take no. of trees = 500
• On varying the maxdepth from 1 to 11, we obtained the second plot on the right. The test accuracy seems to be maxing out at maxdepth=9.
• Finally, we will try to find the best value for the no. of predictors to take for random forest. On varying the no. of predictors from 1 to 4, we obtain the third plot on the right. Hence we will take mtry=4 for our random forest model.

Trees and Ensemble Methods

• On running our models with the parameters decided in the previous slide, we obtain the metrics given on the table in the right.
• Clearly, Gradient Boosting is performing the best with Test Accuracy of 92.02% (compared to 90.2% baseline accuracy). The F1 Score obtained is 0.48
• From the confusion matrix on the right, Precision = 0.65 and Recall = 0.38 which is definitely an improvement over the baseline model.
• On plotting the feature importance for the Gradient Boosted Decision Tree Model, we get the third figure on the right.
• Clearly, nearly all the variables have similar importances with no single dominant variable. Among the 4 variable, miss_distance seems to be the most important feature.

Conclusion

1. The models being used by us do not seem to be extremely good at predicting whether an NEO will be hazardous or not. We have achieved only modest gains over the baseline accuracy
2. Only 4 variables are being used to predict whether an NEO is hazardous or not (miss_distance, absolute magnitude, est_diameter_max and relative_velocity).
3. None of the 4 selected variables seem to be heavily impacting the target variable. All the 4 variables seem to have similar values for feature importance in the Gradient Boosted Tree Model. Other classification models too did not give a wide variation in relative importance of features.

Conclusion

1. Given the aforementioned point, it seems reasonable to expect the findings of Point 1 as the four variables do not seem to be great predictors. Our best model obtained a test accuracy of 92.015% and an F1 score of around 0.48
2. miss_distance seems to be the most important feature as per our best model (Gradient Boosted Decision Tree). However, as per Logistic Regression, this is est_diameter_max and as per Naive Bayes it is absolute_magnitude. This variance probably probably suggests that no strong relationship exists and all of the variables are nearly similar in importance.

Next Steps

• Find a larger dataset with more even dependent variable distribution
• Use more variables
• Limitation: many asteroids were listed multiple times

Questions?