[SOLVED] CS6035 Machine Learning Summer26

Learning Goals of this Project:

• Learning basic Pandas Dataframe manipulations
• Learning more about Machine Learning (ML) Classification models and how they are used in a Cybersecurity context.
• Learning about basic data pipelines and transformations.
• Learning how to write and use unit tests when developing Python code.

Important Highlights

• You can do this project on your host, you do not need to use the VM.
• Please see the Setup page for videos and instructions about project setup.
• Keep the VM around for the final project (Summer 24), Web Security.
• Please watch the provided videos below to see how to setup your environment, we can’t provide broad support here
• There are only 25 submissions allowed! This is because Gradescope is a limited resource. It’s improper to test your code against Gradescope.
• We have provided a local testing suite, be sure to pass that completely before you submit to Gradescope.

Important Reference Materials:

• NumPy Documentation
• Pandas Documentation
• Scikit-learn Documentation

Project Overview Video

This is a 16 minute video by the project creator, it covers project concepts.https://www.youtube.com/embed/kYoQiAamIpQ?si=HRUz0RA4-8IuDhdn

There are other videos on the Setup page that cover installation and other subjects.

BACKGROUND

Many of the Projects in CS6035 are focused on offensive security tasks. These are related to Red Team activities/tasks that many of us may associate with cybersecurity. This project will be focused on defensive security tasks, which are usually considered Blue Team activities that are done by many corporate teams.

Historically, many defensive security professionals have investigated malicious activity, files, and code. They investigate these to create patterns (often called signatures) that can be used to detect (and prevent) malicious activity, files, and code when that pattern is used again. What this means is that these simple methods only were effective on known threats.

This approach was relatively effective in preventing known malware from infecting systems, but it did nothing to protect against novel attacks. As attackers became more sophisticated, they learned to tweak or simply encode their malicious activity, files, or code to avoid detection from these simple pattern matching detections.

With this background information, it would be nice if a more general solution could give a score to activity, files, and code that pass through corporate systems every day. This solution would inform the security team that while a certain pattern may not exactly fit a signature of known malicious activity, files, or code it appears to be very similar to examples that were seen in the past that were malicious.

Luckily machine learning models can do exactly that if provided with proper training data! Thus, it is no surprise that one of the most powerful tools in the hands of defensive cybersecurity professionals is Machine Learning. Modern detection systems usually use a combination of machine learning models and pattern matching (regular expressions) to detect and prevent malicious activity on networks and devices.

This project will focus on teaching the fundamentals of data analysis and building/testing your own machine learning models in python. You’ll be using the open source libraries Pandas and Scikit-Learn.

Cybersecurity Machine Learning Careers and Trends

• Machine learning in cybersecurity is a growing field. The area was considered among top trends by McKinsey in 2022.
• In the CompTIA State of Cybersecurity 2024 it says last year there were 660,000 unfilled Cybersecurity positions. Also in the section titled Product: AI Drives the Cybersecurity Product Set to New Heights they note that 56% of respondents use AI and Machine Learning for Cybersecurity.

Additional Information

• ML in Cybersecurity – Crowdstrike
• AI for Cybersecurity – IBM
• Future of Cybersecurity and AI – Deloitte

Table of contents

• FAQ
• Setup
• Task 1
• Task 2
• Task 3
• Task 4
• Task 5
• Submissions
• Optional Notebooks
• Video Tasks

Task 1 (15 points)

For the first task, let’s get familiar with some pandas basics. pandas is a Python library that deals with Dataframes, which you can think of as a Python class that handles tabular data. In the real world, you would create graphics and other visuals to better understand the dataset you are working with. You would also use plotting tools like PowerBi, Tableau, Data Studio, and Matplotlib. This step is generally known as Exploratory Data Analysis. Since we are using an autograder for this class, we will skip the plotting for this project.

For this task, we have released a local test suite. If you are struggling to understand the expected input and outputs for a function, please set up the test suite and use it to debug your function. Please note that the return lines for the provided skeleton functions are placeholders for the data types that the tests are expecting.

It’s critical you pass all tests locally before you submit to Gradescope for credit. Do not use Gradescope for debugging.

Theory

In this Task, we’re not yet getting into theory. It’s more nuts and bolts – you will learn the basics of pandas. pandas dataframes are something of a glorified list of lists, mixed in with a dictionary. You get a table of values with rows and columns, and you can modify the column names and index values for the rows. There are numerous functions built into pandas to let you manipulate the data in the dataframe.

To be clear, pandas is not part of Python, so when you look up docs, you’ll specifically want the official Pydata pandas docs. Note that we linked to the API docs here, this is the core of the docs you’ll be looking at.

You can always get started trying to solve a problem by looking at Stack Overflow posts in Google search results. There you’ll find ideas about how to use the pandas library. In the end, however, you should find yourself in the habit of looking directly at the docs for whichever library you are using, pandas in this case.

For those who might need a concrete example to get started, here’s how you would take a pandas dataframe column and return the average of its values:

import pandas as pd
# create a dataframe from a Python dict
df = pd.DataFrame({"color":["yellow", "green", "purple", "red"], "weight":[124,4.56,384,-2]})
df # shows the dataframe

Note that the column names are [“color”,”weight”] while the index is [0,1,2,3…] where […] the brackets denote a list.

Now that we have created a dataframe, we can find the average weight by summing the values under ‘weight’ and dividing them by the sum:

average = df['weight'].sum() / len(df['weight'])  
average # if you put a variable as the last line, the variable is printed
127.63999999999999

Note: In the example above, we’re not paying attention to rounding, you will need to round your answers to the precision asked for in each Task.

Also note, we are using slightly older versions of the pandas, Python and other libraries so be sure to look at the docs for the appropriate library version. Often there’s a drop-down at the top of docs sites to select the older version.

Refer to the Submissions page for details about submitting your work.

Useful Links:

• pandas documentation — pandas documentation (pydata.org)
• What is Exploratory Data Analysis? – IBM
• Top Data Visualization Tools – KDnuggets

Deliverables:

• Complete the functions in task1.py
• For this task we have released a local test suite please set that up and use it to debug your function.
• Submit task1.py to gradescope

Instructions:

The Task1.py file has function skeletons that you will complete with Python code, mostly using the pandas library. The goal of each of these functions is to give you familiarity with the pandas library and some general Python concepts like classes, which you may not have seen before. See information about the function’s inputs, outputs, and skeletons below.

Example of Solving Task1 subtasks:

Subtask Description In this function you will take a dataset, a random state, and a number n. You will return n sampled rows from the dataset using the random state to ensure reproducability.

Useful Resource
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sample.html

Inputs

• dataset – a pandas DataFrame.
• n – integer, number of rows to return.
• random_state – integer, seed value to get repeatable results

Outputs

• A pandas DataFrame containing n randomly selected rows.

Function Skeleton

def get_random_sample(dataset: pd.DataFrame, n: int, random_state: int) -> pd.DataFrame:
    return pd.DataFrame()

Expected Result

def get_random_sample(dataset: pd.DataFrame, n: int, random_state: int) -> pd.DataFrame:
    return dataset.sample(n=n,random_state=random_state)

Explaination Reviewing the link in the Useful Resource we can see pandas has a DataFrame.sample method that can be used to sample from a pd.DataFrame Class and returns a pd.DataFrame class. The task Description mentions that you will use n and random_state to return the expected pd.DataFrame object looking at the method it also uses the same naming scheme for those variables so we can use the dataset variable (which is a pd.DataFrame), the DataFrame.sample method and the input parameters passed into our skeleton to return the expected result. This result could then be tested in the notebook and compared with an expected result given sample data and in the unittests which would also pass in sample inputs to the function and compare the function outputs to the expected outputs.

Local Test Dataset Information

For this task the local test dataset we are using a very simple example dataset, which contains 10 rows and 6 columns of data related to network traffic.

In this task we will guide you through splitting datasets into train and test sets as well as preprocessing datasets using scikit-learn encoders, scalers and dimensionality reduction techniques.

Table of contents

1. find_data_type
2. set_index_col
3. reset_index_col
4. set_col_type
5. make_DF_from_2d_array
6. sort_DF_by_column
7. drop_NA_cols
8. drop_NA_rows
9. make_new_column
10. left_merge_DFs_by_column
11. simpleClass
12. find_dataset_statistics

find_data_type

In this function you will take a dataset and the name of a column in it. You will return the column’s data type.

Useful Resources

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dtypes.html

INPUTS

• dataset – a pandas DataFrame that contains some data
• column_name – a Python string (str)

OUTPUTS

np.dtype – data type of the column

Function Skeleton

def find_data_type(dataset:pd.DataFrame,column_name:str) -> np.dtype:
    return np.dtype()

set_index_col

In this function you will take a dataset and a series and set the index of the dataset to be the series

Useful Resources

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Index.html

INPUTS

• dataset – a pandas DataFrame that contains some data
• index – a pandas series that contains an index for the dataset

OUTPUTS

a pandas DataFrame indexed by the given index series

Function Skeleton

def set_index_col(dataset:pd.DataFrame,index:pd.Series) -> pd.DataFrame:
    return pd.DataFrame()

reset_index_col

In this function you will take a dataset with an index already set and reindex the dataset from 0 to n-1, where n is the number of rows in the dataset, dropping the old index

Useful Resources

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.reset_index.html

INPUTS

• dataset – a pandas DataFrame that contains some data

OUTPUTS

a pandas DataFrame indexed from 0 to n-1

Function Skeleton

def reset_index_col(dataset:pd.DataFrame) -> pd.DataFrame:
    return pd.DataFrame()

set_col_type

In this function you will be given a DataFrame, column name and column type. You will edit the dataset to take the column name you are given and set it to be the type given in the input variable

Useful Resources

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.astype.html

INPUTS

• dataset – a pandas DataFrame that contains some data
• column_name – a string containing the name of a column
• new_col_type – a Python type to change the column to

OUTPUTS

a pandas DataFrame with the column in column_name changed to the type in new_col_type

Function Skeleton

# Set astype (string, int, datetime)
def set_col_type(dataset:pd.DataFrame,column_name:str,new_col_type:type) -> pd.DataFrame:
    return pd.DataFrame()

make_DF_from_2d_array

In this function you will take data in an array as well as column and row labels and use that information to create a pandas DataFrame

Useful Resources

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html

INPUTS

• array_2d – a 2 dimensional numpy array of values
• column_name_list – a list of strings holding column names
• index – a pandas series holding the row index’s

OUTPUTS

a pandas DataFrame with columns set from column_name_list, row index set from index and data set from array_2d

Function Skeleton

# Take Matrix of numbers and make it into a DataFrame with column name and index numbering
def make_DF_from_2d_array(array_2d:np.array,column_name_list:list[str],index:pd.Series) -> pd.DataFrame:
    return pd.DataFrame()

sort_DF_by_column

In this function, you are given a dataset and column name. You will return a sorted dataset (sorting rows by the value of the specified column) either in descending or ascending order, depending on the value in the descending variable.

Useful Resources

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sort_values.html

INPUTS

• dataset – a pandas DataFrame that contains some data
• column_name – a string that contains the column name to sort the data on
• descending – a boolean value (True or False) for if the column should be sorted in descending order

OUTPUTS

a pandas DataFrame sorted by the given column name and in descending or ascending order depending on the value of the descending variable

Function Skeleton

# Sort DataFrame by values
def sort_DF_by_column(dataset:pd.DataFrame,column_name:str,descending:bool) -> pd.DataFrame:
    return pd.DataFrame()

drop_NA_cols

In this function you are given a DataFrame. You will return a DataFrame with any columns containing NA values dropped (meaning the returned DataFrame will only contain columns from the input DataFrame that do not contain any NA values).

Useful Resources

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html

INPUTS

• dataset – a pandas DataFrame that contains some data

OUTPUTS

a pandas DataFrame with any columns that contain an NA value dropped

Function Skeleton

# Drop NA values in DataFrame Columns 
def drop_NA_cols(dataset:pd.DataFrame) -> pd.DataFrame:
    return pd.DataFrame()

Example

Input DataFrame:

Output DataFrame:

drop_NA_rows

In this function you are given a DataFrame you will return a DataFrame with any rows containing NA values dropped

Useful Resources

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html

INPUTS

• dataset – a pandas DataFrame that contains some data

OUTPUTS

a pandas DataFrame with any rows that contain an NA value dropped

Function Skeleton

def drop_NA_rows(dataset:pd.DataFrame) -> pd.DataFrame:
    return pd.DataFrame()

make_new_column

This function adds a new column to a DataFrame using a provided list of values, where each value corresponds to a row in the dataset. The new column is named according to the specified new_column_name.

Useful Resources

• Adding Columns in Pandas

INPUTS

• dataset – A pandas DataFrame containing existing data.
• new_column_name – A string specifying the name of the new column to create.
• new_column_value – A list of values where each element represents the value for the new column in the corresponding row of the DataFrame. The length of this list must match the number of rows in the dataset.

OUTPUTS

A pandas DataFrame with the new column added. The new column, named new_column_name, contains the values from the new_column_value list in the order they are provided. Each row’s value in the new column corresponds to the element at the same index in the list.

Function Skeleton

def make_new_column(dataset:pd.DataFrame,new_column_name:str,new_column_value:list) -> pd.DataFrame:
    return pd.DataFrame()

left_merge_DFs_by_column

In this function you are given 2 datasets and the name of a column with which you will left join them on using the pandas merge method. The left dataset is dataset1 right dataset is dataset2, for example purposes.

Useful Resources

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.merge.html https://stackoverflow.com/questions/53645882/pandas-merging-101

INPUTS

• left_dataset – a pandas DataFrame that contains some data
• right_dataset – a pandas DataFrame that contains some data
• join_col_name – a string containing the column name to join the two DataFrames on

OUTPUTS

a pandas DataFrame containing the two datasets left joined together on the given column name

Function Skeleton

def left_merge_DFs_by_column(left_dataset:pd.DataFrame,right_dataset:pd.DataFrame,join_col_name:str) -> pd.DataFrame:
    return pd.DataFrame()

simpleClass

This project will require you to work with Python Classes. If you are not familiar with them we suggest learning a bit more about them.

You will take the inputs into the class initialization and set them as instance variables (of the same name) in the Python class.

Useful Resources

https://www.w3schools.com/python/python_classes.asp

INPUTS

• length – an integer
• width – an integer
• height – an integer

OUTPUTS

None, just setup the init method in the class.

Function Skeleton

class simpleClass():
    def __init__(self, length:int, width:int, height:int):
        pass

find_dataset_statistics

Now that you have learned a bit about pandas DataFrames, we will use them to generate some simple summary statistics for a DataFrame. You will be given the dataset as an input variable, as well as a column name for a column in the dataset that serves as a label column. This label column contains binary values (0 and 1) that you also summarize, and also the variable to predict.

In this context:

• 0 represents a “negative” sample (e.g. if the column is IsAVirus and we think it is false)
• 1 represents a “positive” sample (e.g. if the column is IsAVirus and we think it is true)

This type of binary classification is common in machine learning tasks where we want to be able to predict the field. An example of where this could be useful would be if we were looking at network data, and the label column was IsVirus. We could then analyze the network data of Georgia Tech services and predict if incoming files look like a virus (and if we should alert the security team).

Useful Resources

• https://www.learndatasci.com/glossary/binary-classification/
• https://developers.google.com/machine-learning/crash-course/framing/ml-terminology

INPUTS

• dataset – a pandas DataFrame that contains some data
• label_col – a string containing the name of the label column

OUTPUTS

• n_records (int) – the number of rows in the dataset
• n_columns (int) – the number of columns in the dataset
• n_negative (int) – the number of “negative” samples in the dataset (the argument label column equals 0)
• n_positive (int) – the number of “positive” samples in the dataset (the argument label column equals 1)
• perc_positive (int) – the percentage (out of 100%) of positive samples in the dataset; truncate anything after the decimal

Hint: Consider using the int function to type cast decimals

Function Skeleton

def find_dataset_statistics(dataset:pd.DataFrame,label_col:str) -> tuple[int,int,int,int,int]:
  n_records = #TODO
  n_columns = #TODO
  n_negative = #TODO
  n_positive = #TODO
  perc_positive = #TODO
  return n_records,n_columns,n_negative,n_positive,perc_positive

def tts(  dataset: pd.DataFrame,
          label_col: str, 
          test_size: float,
          should_stratify: bool,
          random_state: int) -> tuple[pd.DataFrame,pd.DataFrame,pd.Series,pd.Series]:
    # TODO
    return train_features,test_features,train_labels,test_labels

def double_height(dataframe:pd.DataFrame):
    return dataframe["height"] * 2
    
def half_height(dataframe:pd.DataFrame):
    return dataframe["height"] / 2
    
feature_engineering_functions = {"double_height":double_height,"half_height":half_height}

def __init__(self, 
                 one_hot_encode_cols:list[str],
                 min_max_scale_cols:list[str],
                 n_components:int,
                 feature_engineering_functions:dict
                 ):
        # TODO: Add any instance variables you may need to make your functions work
        return

def one_hot_encode_columns_train(self,train_features:pd.DataFrame) -> pd.DataFrame:
    one_hot_encoded_dataset = pd.DataFrame()
    return one_hot_encoded_dataset
def one_hot_encode_columns_test(self,test_features:pd.DataFrame) -> pd.DataFrame:
    one_hot_encoded_dataset = pd.DataFrame()
    return one_hot_encoded_dataset

X_std = (X - X.min(axis=0)) / (X.max(axis=0) - X.min(axis=0))
X_scaled = X_std * (max - min) + min

def min_max_scaled_columns_train(self,train_features:pd.DataFrame) -> pd.DataFrame:
        min_max_scaled_dataset = pd.DataFrame()
        return min_max_scaled_dataset
    def min_max_scaled_columns_test(self,test_features:pd.DataFrame) -> pd.DataFrame:
        min_max_scaled_dataset = pd.DataFrame()
        return min_max_scaled_dataset

def pca_train(self,train_features:pd.DataFrame) -> pd.DataFrame:
     # TODO: Read the function description in https://github.gatech.edu/pages/cs6035-tools/cs6035-tools.github.io/Projects/Machine_Learning/Task2.html and implement the function as described
     pca_dataset = pd.DataFrame()
     return pca_dataset

 def pca_test(self,test_features:pd.DataFrame) -> pd.DataFrame:
     # TODO: Read the function description in https://github.gatech.edu/pages/cs6035-tools/cs6035-tools.github.io/Projects/Machine_Learning/Task2.html and implement the function as described
     pca_dataset = pd.DataFrame()
     return pca_dataset

import pandas as pd
def double_height(dataframe: pd.DataFrame) -> pd.Series:
    return dataframe["height"] * 2
    
def half_height(dataframe: pd.DataFrame) -> pd.Series:
    return dataframe["height"] / 2
    
example_feature_engineering_functions = {
    "double_height": double_height, # Note that functions in python can be passed around and used just like data!
    "half_height": half_height
}

# and the class may be been created like this...
# preprocessor = PreprocessDataset(..., feature_engineering_functions=example_feature_engineering_functions, ...)

def feature_engineering_train(self,train_features:pd.DataFrame) -> pd.DataFrame:
    feature_engineered_dataset = pd.DataFrame()
    return feature_engineered_dataset
def feature_engineering_test(self,test_features:pd.DataFrame) -> pd.DataFrame:
    feature_engineered_dataset = pd.DataFrame()
    return feature_engineered_dataset

def preprocess_train(self,train_features:pd.DataFrame) -> pd.DataFrame:
  preprocessed_dataset = pd.DataFrame()
  return train_features

def preprocess_test(self,test_features:pd.DataFrame) -> pd.DataFrame:
  preprocessed_dataset = pd.DataFrame()
  return test_features

def __init__(self, 
             random_state: int,
             init: str, 
             n_init: int,
             max_iter: int,
             algorithm: str,
             tol: float
            ):
  # TODO: Add any state variables you may need to make your functions work
  pass

def kmeans_get_n_clusters(self, train_features:pd.DataFrame) -> int:
    k = int
    return k

def kmeans_train(self, train_features:pd.DataFrame, k:int=None) -> list:
    cluster_ids = list()
    return cluster_ids

def kmeans_test(self, test_features:pd.DataFrame) -> list:
        cluster_ids = list()
        return cluster_ids

Task 4 (25 points)

Now let’s try a few supervised classification models, we have chosen a few commonly used models for you to use here, but there are many options. In the real world, specific algorithms may fit a specific dataset better than other algorithms.

You won’t be doing any hyperparameter tuning yet, so you can better focus on writing the basic code. You will:

• Train a model using the training set.
• Predict on the training/test sets.
• Calculate performance metrics.
• Return a ModelMetrics object and trained scikit-learn model from each model function.

Important Note on Feature Selection: You should ONLY use RFE (Recursive Feature Elimination) for determining feature importance of your Logistic Regression model. Do NOT use RFE for any tree-based models (Decision Tree, Random Forest, or Gradient Boosting). For tree-based models, use their built-in feature importance values instead.

Useful Links:

• scikit-learn: machine learning in Python — scikit-learn 1.2.1 documentation
• Training and Test Sets – Machine Learning – Google Developers
• Bias–variance tradeoff – Wikipedia
• Overfitting – Wikipedia
• An Introduction to Classification in Machine Learning – builtin
• Classification in Machine Learning: An Introduction – DataCamp

Deliverables:

• Complete the functions and methods in task4.py
• For this task we have released a local test suite please set that up and use it to debug your function.
• Submit task4.py to Gradescope when you pass all local tests. Refer to the Submissions page for details.

  Local Test Dataset Information

  For this task the local test dataset we are using is the NATICUSdroid dataset, which contains 86 columns of data related to android permissions used by benign and malicious Android applications released between 2010 and 2019. For more information such as the introductory paper and the Citations/Acknowledgements you can view the dataset site in the UCI ML repository. If you look at the online poster for the paper that the dataset creators wrote from their research, they trained a variety of different models including Random Forest, Logistic Regression and XGBoost and calculated a variety of metrics related to training and detection performance. In this task we will guide you through training ML models and calculating performance metrics to compare the predictive abilities of different models.

Instructions:

The Task4.py File has function skeletons that you will complete with Python code (mostly using the pandas and scikit-learn libraries).

The goal of each of these functions is to give you familiarity with the applied concepts of training a model, using it to score records and calculating performance metrics for it. See information about the function inputs, outputs and skeletons below.

Table of contents

1. ModelMetrics
2. calculate_naive_metrics
3. calculate_logistic_regression_metrics
4. calculate_decision_tree_metrics
5. calculate_gradient_boosting_metrics

ModelMetrics

• In order to simplify the autograding we have created a class that will hold the metrics and feature importances for a model you trained.
• You should not modify this class but are expected to use it in your return statements.
• This means you put your training and test metrics dictionaries and feature importance DataFrames inside a ModelMetrics object for the autograder to handle. This is for each of the Logistic Regression, Gradient Boosting and Random Forest models you will create.
• You do not need to return a feature importance DataFrame in the ModelMetrics value for the naive model you will create, just return None in that position of the return statement, as the given code does.

calculate_naive_metrics

A Naive model is a very simple model/prediction that can help to frame how well a more sophisticated model is doing. At best, such a model has random competence at predicting things. At worst, it’s wrong all the time.

Since a naive model is incredibly basic (often a constant or randomly selected result), we can expect that any more sophisticated model that we train should outperform it. If the naive Model beats our trained model, it can mean that additional data (rows or columns) is needed in the dataset to improve our model. It can also mean that the dataset doesn’t have a strong enough signal for the target we want to predict.

In this function, you’ll implement a simple model that always predicts a constant (function-provided) number, regardless of the input values. Specifically, you’ll use a given constant integer, provided as the parameter naive_assumption, as the model’s prediction. This means the model will always output this constant value, without considering the actual data. Afterward, you will calculate four metrics—accuracy, recall, precision, and F1-score—for both the training and test datasets.

[1] Refer to the resources below.

Useful Resources

• https://machinelearningmastery.com/how-to-develop-and-evaluate-naive-classifier-strategies-using-probability/
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score

INPUTS

• train_features – a dataset split by a function similar to the tts function you created in task2
• test_features – a dataset split by a function similar to the tts function you created in task2
• train_targets – a dataset split by a function similar to the tts function you created in task2
• test_targets – a dataset split by a function similar to the tts function you created in task2
• naive_assumption – an integer that should be used as the result from the naive model you will create

OUTPUTS

A completed ModelMetrics object with a training and test metrics dictionary with each one of the metrics rounded to 4 decimal places

Function Skeleton

def calculate_naive_metrics(train_features:pd.DataFrame, test_features:pd.DataFrame, train_targets:pd.Series, test_targets:pd.Series, naive_assumption:int) -> ModelMetrics:
    train_metrics = {
        "accuracy" : 0,
        "recall" : 0,
        "precision" : 0,
        "fscore" : 0
        }
    test_metrics = {
        "accuracy" : 0,
        "recall" : 0,
        "precision" : 0,
        "fscore" : 0
        }
    naive_metrics = ModelMetrics("Naive",train_metrics,test_metrics,None)
    return naive_metrics

calculate_logistic_regression_metrics

Important Note on Feature Selection: RFE (Recursive Feature Elimination) should ONLY be used for Logistic Regression. Do NOT use RFE for any tree-based models (Decision Tree, Random Forest, or Gradient Boosting) – use their built-in feature importance values instead.

A logistic regression model is a simple and more explainable statistical model that can be used to estimate the probability of an event (log-odds). At a high level, a logistic regression model uses data in the training set to estimate a column’s weight in a linear approximation function. Conceptually this is similar to estimating m for each column in the line formula you probably know well from geometry: y = m*x + b. If you are interested in learning more, you can read up on the math behind how this works. For this project, we are more focused on showing you how to apply these models, so you can simply use a scikit-learn Logistic Regression model in your code.

For this task use scikit-learn’s LogisticRegression class and complete the following subtasks:

• Train a Logistic Regression model (initialized using the kwargs passed into the function)
• Predict scores for training and test datasets and calculate the 7 metrics listed below for the training and test datasets using predictions from the fit model. (All rounded to 4 decimal places)
  • accuracy
  • recall
  • precision
  • fscore
  • false positive rate (fpr)
  • false negative rate (fnr)
  • Area Under the Curve of Receiver Operating Characteristics Curve (roc_auc)
• Use RFE to select the top n_feat_importance features
• Train a Logistic Regression model using these selected features (initialized using the kwargs passed into the function)
• Create a Feature Importance DataFrame from the model trained on the top n_feat_importance features:
  • Use the top N features (where N is configured via n_feat_importance input variable) and sort by absolute value of the coefficient from biggest to smallest.
  • Make sure you use the same feature and importance column names as set in ModelMetrics in feat_name_col [Feature] and imp_col [Importance].
  • Round the importances to 4 decimal places (do this step after you have sorted by Importance)
  • Reset the index to 0->(N-1). You can do this the same way you did in task1.

NOTE: Make sure you use the predicted probabilities for roc auc

Useful Resources

• https://stats.libretexts.org/Bookshelves/Introductory_Statistics/OpenIntro_Statistics_(Diez_et_al)./08%3A_Multiple_and_Logistic_Regression/8.04%3A_Introduction_to_Logistic_Regression
• https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score
• https://en.wikipedia.org/wiki/Confusion_matrix
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html

INPUTS

The first 4 are similar to the tts function you created in Task 2:

• train_features – a Pandas Dataframe with training features
• test_features – a Pandas Dataframe with test features
• train_targets – a Pandas Dataframe with training targets
• test_targets – a Pandas Dataframe with test targets
• n_feat_importance – an int with how many features to return in the feature importance dataframe
• logreg_kwargs – a dictionary with keyword arguments that can be passed directly to the scikit-learn Logistic Regression class

OUTPUTS

• A completed ModelMetrics object with a training and test metrics dictionary with each one of the metrics rounded to 4 decimal places
• A scikit-learn Logistic Regression model object fit on the training set

Function Skeleton

def calculate_logistic_regression_metrics(train_features:pd.DataFrame, test_features:pd.DataFrame, train_targets:pd.Series, test_targets:pd.Series, n_feat_importance: int, logreg_kwargs) -> tuple[ModelMetrics,LogisticRegression]:
    model = LogisticRegression()
    train_metrics = {
        "accuracy" : 0,
        "recall" : 0,
        "precision" : 0,
        "fscore" : 0,
        "fpr" : 0,
        "fnr" : 0,
        "roc_auc" : 0
        }
    test_metrics = {
        "accuracy" : 0,
        "recall" : 0,
        "precision" : 0,
        "fscore" : 0,
        "fpr" : 0,
        "fnr" : 0,
        "roc_auc" : 0
        }

    log_reg_importance = pd.DataFrame()
    log_reg_metrics = ModelMetrics("Logistic Regression",train_metrics,test_metrics,log_reg_importance)

    return log_reg_metrics,model

Example of Feature Importance DataFrame (with n_feat_importance=10)

calculate_decision_tree_metrics

Important Note on Feature Selection: RFE (Recursive Feature Elimination) should ONLY be used for Logistic Regression. Do NOT use RFE for any tree-based models (Decision Tree, Random Forest, or Gradient Boosting). For tree-based models, use their built-in feature importance values instead.

A Decision Tree (DT) is a supervised learning algorithm used for both classification and regression tasks. It works by recursively splitting the data into subsets based on the feature that results in the best separation of classes, typically measured using Gini impurity or entropy. Decision trees are interpretable, as the learned model can be visualized as a flowchart-like structure.

If you are interested in learning more, you can read up on the math behind how decision trees work.

For this project, we are more focused on showing you how to apply these models, so you can simply use a scikit-learn DecisionTreeClassifier in your code.

For this task, use scikit-learn’s DecisionTreeClassifier class and complete the following subtasks:

• Train a DT model (initialized using the kwargs passed into the function).
• Predict scores for training and test datasets and calculate the 7 metrics listed below for the training and test datasets using predictions from the fit model. (All rounded to 4 decimal places)
  • accuracy
  • recall
  • precision
  • fscore
  • false positive rate (fpr)
  • false negative rate (fnr)
  • Area Under the Curve of Receiver Operating Characteristics Curve (roc_auc)
• Create a Feature Importance DataFrame from the trained model:
  • Do Not Use RFE for feature selection
  • Use the top N (where N is configured via n_feat_importance input variable) features selected by the built in method (sorted from biggest to smallest)
  • Make sure you use the same feature and importance column names as ModelMetrics shows in feat_name_col [Feature] and imp_col [Importance]
  • Round the importances to 4 decimal places (do this step after you have sorted by Importance)
  • Reset the index to 0->(N-1) you can do this the same way you did in task1

NOTE: Make sure you use the predicted probabilities for roc_auc.

Useful Resources

• https://scikit-learn.org/stable/modules/tree.html
• https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score
• https://en.wikipedia.org/wiki/Confusion_matrix
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html

INPUTS

The first 4 are similar to the tts function you created in Task 2:

• train_features – a Pandas DataFrame with training features
• test_features – a Pandas DataFrame with test features
• train_targets – a Pandas DataFrame with training targets
• test_targets – a Pandas DataFrame with test targets
• n_feat_importance – an int with how many features to return in the feature importance dataframe
• dt_kwargs – a dictionary with keyword arguments that can be passed directly to the scikit-learn DecisionTreeClassifier class

OUTPUTS

• A completed ModelMetrics object with a training and test metrics dictionary with each one of the metrics rounded to 4 decimal places
• A scikit-learn DecisionTreeClassifier model object fit on the training set

Function Skeleton

def calculate_decision_tree_metrics(train_features: pd.DataFrame, test_features: pd.DataFrame, train_targets: pd.Series, test_targets: pd.Series, n_feat_importance: int, dt_kwargs) -> tuple[ModelMetrics, DecisionTreeClassifier]:
    model = DecisionTreeClassifier(**dt_kwargs)
    train_metrics = {
        "accuracy": 0,
        "recall": 0,
        "precision": 0,
        "fscore": 0,
        "fpr": 0,
        "fnr": 0,
        "roc_auc": 0
    }
    test_metrics = {
        "accuracy": 0,
        "recall": 0,
        "precision": 0,
        "fscore": 0,
        "fpr": 0,
        "fnr": 0,
        "roc_auc": 0
    }

    dt_importance = pd.DataFrame()
    dt_metrics = ModelMetrics("Decision Tree", train_metrics, test_metrics, dt_importance)

    return dt_metrics, model

Example of Feature Importance DataFrame (with n_feat_importance=5)

calculate_gradient_boosting_metrics

Important Note on Feature Selection: RFE (Recursive Feature Elimination) should ONLY be used for Logistic Regression. Do NOT use RFE for any tree-based models (Decision Tree, Random Forest, or Gradient Boosting). For tree-based models, use their built-in feature importance values instead.

A Gradient Boosted model is more complex than the Naive and Logistic Regression models and similar in structure to the Decision Tree model you trained. A Gradient Boosted model expands on the tree-based model by using its additional trees to predict the errors from the previous tree. For this project, we are more focused on showing you how to apply these models, so you can simply use the scikit-learn Gradient Boosted Model in your code.

For this task use scikit-learn’s Gradient Boosting Classifier class and complete the following subtasks:

• Train a Gradient Boosted model (initialized using the kwargs passed into the function)
• Predict scores for training and test datasets and calculate the 7 metrics listed below for the training and test datasets using predictions from the fit model. (All rounded to 4 decimal places)
  • accuracy
  • recall
  • precision
  • fscore
  • false positive rate (fpr)
  • false negative rate (fnr)
  • Area Under the Curve of Receiver Operating Characteristics Curve (roc_auc)
• Create a Feature Importance DataFrame from the trained model:
  • Do Not Use RFE for feature selection
  • Use the top N (where N is configured via n_feat_importance input variable) features selected by the built in method (sorted from biggest to smallest)
  • Make sure you use the same feature and importance column names as ModelMetrics shows in feat_name_col [Feature] and imp_col [Importance]
  • round the importances to 4 decimal places (do this step after you have sorted by Importance)
  • Reset the index to 0->(N-1) you can do this the same way you did in task1

NOTE: Make sure you use the predicted probabilities for roc auc

Refer to the Submissions page for details about submitting your work.

Useful Resources

• https://blog.dataiku.com/tree-based-models-how-they-work-in-plain-english
• https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score
• https://en.wikipedia.org/wiki/Confusion_matrix
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html

INPUTS

• train_features – a dataset split by a function similar to the tts function you created in task2
• test_features – a dataset split by a function similar to the tts function you created in task2
• train_targets – a dataset split by a function similar to the tts function you created in task2
• test_targets – a dataset split by a function similar to the tts function you created in task2
• n_feat_importance – an int with how many features to return in the feature importance dataframe
• gb_kwargs – a dictionary with keyword arguments that can be passed directly to the scikit-learn GradientBoostingClassifier class

OUTPUTS

• A completed ModelMetrics object with a training and test metrics dictionary with each one of the metrics rounded to 4 decimal places
• An scikit-learn Gradient Boosted model object fit on the training set

Function Skeleton

def calculate_gradient_boosting_metrics(train_features:pd.DataFrame, test_features:pd.DataFrame, train_targets:pd.Series, test_targets:pd.Series, n_feat_importance: int, gb_kwargs) -> tuple[ModelMetrics,GradientBoostingClassifier]:
    model = GradientBoostingClassifier()
    train_metrics = {
        "accuracy" : 0,
        "recall" : 0,
        "precision" : 0,
        "fscore" : 0,
        "fpr" : 0,
        "fnr" : 0,
        "roc_auc" : 0
        }
    test_metrics = {
        "accuracy" : 0,
        "recall" : 0,
        "precision" : 0,
        "fscore" : 0,
        "fpr" : 0,
        "fnr" : 0,
        "roc_auc" : 0
        }

    gb_importance = pd.DataFrame()
    gb_metrics = ModelMetrics("Gradient Boosting",train_metrics,test_metrics,gb_importance)

    return gb_metrics,model

Example of Feature Importance DataFrame (with n_feat_importance=10)

Task 5: Model Training and Evaluation (20 points)

Now that you have written functions for different steps of the model-building process, you will put it all together. You will write code that trains a model with hyperparameters you determine (you should do any tuning locally or in a notebook, i.e., don’t tune your model in gradescope since the autograder will likely timeout).

• Refer to the Submissions page for details about submitting your work.

Important: Conduct hyperparameter tuning locally or in a separate notebook. Avoid tuning within Gradescope to prevent autograder timeouts.

Develop your own local tests to ensure your code functions correctly before submitting to Gradescope. Do not share these tests with other students.

train_model_return_scores_clamp (ClaMP Dataset)

Instructions (5 points):

This function focuses on training a model using the ClaMP dataset and evaluating its performance on a test set.

1. Input:
   • train_df: A Pandas DataFrame containing the ClaMP training data. This includes the “label” column, which serves as your target variable (0 for benign, 1 for malicious).
   • test_df: A Pandas DataFrame containing the ClaMP test data. The “label” column is intentionally omitted from this set.
2. Model Training:
   • Train a machine learning model using the train_df dataset.
   • You may use any techniques covered in this project.
   • Set a random seed for reproducibility.
   • Perform hyperparameter tuning (if needed) to optimize your model’s performance
     • Tip: putting comments on the ranges you select for hyperparameters will help the graders understand how you chose it
   • Document your hyperparameter tuning strategy (if any) in the document_hyperparameter_tuning_clamp function. if you used grid search or some other automated search then running that function should generate the same parameters you used for your submission
3. Prediction:
   • Use your trained model to predict the probability of malware for each row in the test_df.
   • Output these probabilities as values between 0 and 1. A value closer to 0 indicates a lower likelihood of malware, while a value closer to 1 indicates a higher likelihood.
4. Output:
   • Return a Pandas DataFrame with two columns:
     • index: The index from the input test_df.
     • prob_label_1: The predicted malware probabilities.
5. Evaluation:
   • The autograder will evaluate your predictions using the ROC AUC score.
   • You must achieve a ROC AUC score of 0.9 or higher on the test set to receive full credit.

Local Test Case

For this task we do not provide a local test case for you. One of the Learning Objectives of this project is getting familiar using and creating python unittests. You have used tests we created for you in tasks 1-4 and now will get to practice creating your own.

1. Load the training data you were provided in the Task5 folder from the CSV file.
2. Split that file into a train df and a test df (a common split % to use is 80% data for the train set and 20% for the test set). Review Task 2 and the scikit-learn docs for useful parameters to ensure a balanced split.
3. Make a copy of the test df with the label column removed to mimic the environment your model will see in the AG.
4. Run your train_model_return_scores function on those datasets to get your predictions
5. Run Output Checks:
   • Make sure the number of rows in your predictions matches the expected number of rows (compare with rows of your test set)
   • Check if the returned DataFrame has exactly the columns [“index”, “prob_label_1”]. Hint: DataFrame.columns returns the columns of a pandas DataFrame.
   • Calculate the ROC AUC score between your prob_label_1 and the true class labels (compare with test set) you can use the thresholds described in the evaluation section above to consider your model “passing” this test.

Sample Submission (ClaMP):

Function Skeleton (ClaMP):

import pandas as pd

def document_hyperparameter_tuning_clamp(train_df,test_df):
    """
    Please document the hyperparameter tuning process you used to tune your machine learning model for Task 5.

    You should copy and paste the hyperparameter process you conducted here.

    Place all parameters tuned and values in the hyperparameters dictionary.

    If we run your code and your hyperparameter function, it must generate the same hyperparameters you used for your function.
    
    You do not need to return anything specific, just document how you did your tuning.

    We will not run it in the autograder, it is just an additional check on our end.
    """

    hyperparameters = {}

    return hyperparameters

def train_model_return_scores_clamp(train_df, test_df) -> pd.DataFrame:
    """
    Trains a model on the ClaMP training data and returns predicted probabilities 
    for the test data.

    Args:
        train_df (pd.DataFrame): ClaMP training data with 'label' column.
        test_df (pd.DataFrame): ClaMP test data without 'label' column.

    Returns:
        pd.DataFrame: DataFrame with 'index' and 'prob_label_1' columns.
    """
    # TODO: Implement the model training and prediction logic as described above.
    test_scores = pd.DataFrame()  # Replace with your implementation
    return test_scores

ClaMP Dataset

• The ClaMP (Classification of Malware with PE Headers) dataset is used for malware classification.
• It is based on the header fields of Portable Executable (PE) files.
• Learn more about PE files:
  • Microsoft – PE Format
  • Wikipedia – Portable Executable
• ClaMP Dataset GitHub Repository: https://github.com/urwithajit9/ClaMP
• This project uses the ClaMP_Raw-5184.csv file (55 features).

  train_model_return_scores_unsw (UNSW-NB15 Dataset)

Instructions (10 points):

This function focuses on training a model using the UNSW-NB15 dataset and evaluating its performance on a test set. It will likely require exploring/understanding the dataset, data preprocessing, model selection, and hyperparameter tuning to acheive full credit.

1. Input:
   • train_df: A Pandas DataFrame containing the UNSW-NB15 training data (including the “label” column).
   • test_df: A Pandas DataFrame containing the UNSW-NB15 test data (without the “label” column).
2. Model Training:
   • Train a machine learning model using the train_df.
   • You can use any techniques from this project.
   • Set a random seed for reproducibility.
   • Document your hyperparameter tuning strategy in the document_hyperparameter_tuning_clamp function. if you used grid search or some other automated search then running that function should generate the same parameters you used for your submission
3. Prediction:
   • Predict the probability of label=1 for each row in test_df.
   • Output probabilities between 0 and 1, where values closer to 1 indicate a higher likelihood of being label=1.
4. Output:
   • Return a Pandas DataFrame with two columns:
     • index: The index from the input test_df.
     • prob_label_1: The predicted probabilities of label=1.
5. Evaluation:
   • The autograder will evaluate your predictions using the ROC AUC score.
   • Full Credit (10 points) will be given for 0.76 and above, 5 points for .75 and above and 2.5 points for .55 and above
   • Parameter tuning will likely be necessary to achieve higher scores.

Local Test Case

For this task we do not provide a local test case for you. One of the Learning Objectives of this project is getting familiar using and creating python unittests. You have used tests we created for you in tasks 1-4 and now will get to practice creating your own.

1. Load the training data you were provided in the Task5 folder from the CSV file.
2. Split that file into a train df and a test df (a common split % to use is 80% data for the train set and 20% for the test set). Review Task 2 and the scikit-learn docs for useful parameters to ensure a balanced split.
3. Make a copy of the test df with the label column removed to mimic the environment your model will see in the AG.
4. Run your train_model_return_scores function on those datasets to get your predictions
5. Run Output Checks:
   • Make sure the number of rows in your predictions matches the expected number of rows (compare with rows of your test set)
   • Check if the returned DataFrame has exactly the columns [“index”, “prob_label_1”]. Hint: DataFrame.columns returns the columns of a pandas DataFrame.
   • Calculate the ROC AUC score between your prob_label_1 and the true class labels (compare with test set) you can use the thresholds described in the evaluation section above to consider your model “passing” this test.

Sample Submission (UNSW-NB15):

Function Skeleton (UNSW-NB15):

import pandas as pd

def document_hyperparameter_tuning_unsw(train_df,test_df):
    """
    Please document the hyperparameter tuning process you used to tune your machine learning model for Task 5.

    You should copy and paste the hyperparameter process you conducted here.

    Place all parameters tuned and values in the hyperparameters dictionary.
    
    If we run your code and your hyperparameter function, it must generate the same hyperparameters you used for your function.
    
    You do not need to return anything specific, just document how you did your tuning.

    We will not run it in the autograder, it is just an additional check on our end.
    """

    hyperparameters = {}

    return hyperparameters

def train_model_return_scores_unsw(train_df, test_df) -> pd.DataFrame:
    """
    Trains a model on the UNSW-NB15 training data and returns predicted 
    probabilities for the test data.

    Args:
        train_df (pd.DataFrame): UNSW-NB15 training data with 'label' column.
        test_df (pd.DataFrame): UNSW-NB15 test data without 'label' column.

    Returns:
        pd.DataFrame: DataFrame with 'index' and 'prob_label_1' columns.
    """
    # TODO: Implement the model training and prediction logic as described above.
    test_scores = pd.DataFrame()  # Replace with your implementation
    return test_scores

UNSW-NB15 Dataset

• The UNSW-NB15 dataset was created using the IXIA PerfectStorm tool to simulate real-world network traffic and attack scenarios.
• Dataset Website: https://research.unsw.edu.au/projects/unsw-nb15-dataset
• Dataset Description
• Feature Descriptions
• Note: This project does not use all features or classes from the original UNSW-NB15 dataset.

Additional Resources (optional)

• Hyperparameter Tuning in Scikit-Learn Video

train_model_return_scores_phiusiil (PhiUSIIL Phishing URL Dataset)

Instructions (5 points):

This function focuses on training a model using the PhiUSIIL Phishing URL dataset and evaluating its performance on a test set. It will likely require exploring/understanding the dataset, feature creation, data preprocessing, model selection, and hyperparameter tuning to acheive full credit.

1. Input:
   • train_df: A Pandas DataFrame containing the PhiUSIIL Phishing URL training data (including the “label” column).
   • test_df: A Pandas DataFrame containing the PhiUSIIL Phishing URL test data (without the “label” column).
2. Model Training:
   • You will need to create your own feature engineering functions to pass this task since you are only given a single text column. Review resources for tips on generating features from a URL.
   • Train a machine learning model using the train_df.
   • You can use any techniques from this project.
   • Set a random seed for reproducibility.
   • Document your hyperparameter tuning strategy (if any) in the document_hyperparameter_tuning_clamp function. if you used grid search or some other automated search then running that function should generate the same parameters you used for your submission
3. Prediction:
   • Predict the probability of label=1 for each row in test_df.
   • Output probabilities between 0 and 1, where values closer to 1 indicate a higher likelihood of being label=1.
4. Output:
   • Return a Pandas DataFrame with two columns:
     • index: The index from the input test_df.
     • prob_label_1: The predicted probabilities of label=1.
5. Evaluation:
   • The autograder will evaluate your predictions using the ROC AUC score.
   • Full Credit (5 points) will be given for 0.85 and above, 2.5 points for .8 and above and 1.25 points for .55 and above
   • Parameter tuning will likely be necessary to achieve higher scores.

Local Test Case

For this task we do not provide a local test case for you. One of the Learning Objectives of this project is getting familiar using and creating python unittests. You have used tests we created for you in tasks 1-4 and now will get to practice creating your own.

1. Load the training data you were provided in the Task5 folder from the CSV file.
2. Split that file into a train df and a test df (a common split % to use is 80% data for the train set and 20% for the test set). Review Task 2 and the scikit-learn docs for useful parameters to ensure a balanced split.
3. Make a copy of the test df with the label column removed to mimic the environment your model will see in the AG.
4. Run your train_model_return_scores function on those datasets to get your predictions
5. Run Output Checks:
   • Make sure the number of rows in your predictions matches the expected number of rows (compare with rows of your test set)
   • Check if the returned DataFrame has exactly the columns [“index”, “prob_label_1”]. Hint: DataFrame.columns returns the columns of a pandas DataFrame.
   • Calculate the ROC AUC score between your prob_label_1 and the true class labels (compare with test set) you can use the thresholds described in the evaluation section above to consider your model “passing” this test.

Sample Submission (PhiUSIIL):

Function Skeleton (PhiUSIIL):

import pandas as pd

def document_hyperparameter_tuning_phiusiil(train_df,test_df):
    """
    Please document the hyperparameter tuning process you used to tune your machine learning model for Task 5.
    You should copy and paste the hyperparameter process you conducted here.
    Place all parameters tuned and values in the hyperparameters dictionary.
    If we run your code and your hyperparameter function, it must generate the same hyperparameters you used for your function.
    You do not need to return anything specific, just document how you did your tuning.
    We will not run it in the autograder, it is just an additional check on our end.
    """

    hyperparameters = {}

    return hyperparameters

def train_model_return_scores_phiusiil(train_df, test_df) -> pd.DataFrame:
    """
    Trains a model on the PhiUSIIL Phishing URL training data and returns predicted 
    probabilities for the test data.

    Args:
        train_df (pd.DataFrame): PhiUSIIL Phishing URL training data with 'label' column.
        test_df (pd.DataFrame): PhiUSIIL Phishing URL test data without 'label' column.

    Returns:
        pd.DataFrame: DataFrame with 'index' and 'prob_label_1' columns.
    """
    # TODO: Implement the model training and prediction logic as described above.
    test_scores = pd.DataFrame()  # Replace with your implementation
    return test_scores

PhiUSIIL Phishing URL Dataset

• The PhiUSIIL Phishing URL dataset was created from 235,795 legitimate and phishing urls. For our class we are only giving you the URLs and want you to feature engineer your own predictive features to use in an ML model. This task is open ended meaning you should do your own research to determine what features to generate from a url. You can try to mimic the features the authors created or create your own the only limitation is your features need to be created from the URL alone (without internet access) so more complex strategies that try to scrape the url will not work in the autograder.
• Dataset Website: PhiUSIIL: A diverse security profile empowered phishing URL detection framework based on similarity index and incremental learning
• Note: This project does not use all features or classes from the original PhiUSIIL Phishing URL dataset.

Additional Resources (optional)

• Hyperparameter Tuning in Scikit-Learn Video
• Classification of Malicious URLs Using Machine Learning

Deliverables

1. Local Testing: While it is not a deliverable that we will check in your Gradescope submission, we strongly encourage you to thoroughly test your code locally using the provided datasets. Create your own test sets by splitting the training data as described above.
2. Gradescope Submission: Once you are confident in your solution, submit your task5.py file (containing all functions) to Gradescope.
3.