[SOLVED] CS6035 Machine Learning FALL2026 100%

import pandas as pd
# create a dataframe from a Python dict
df = pd.DataFrame({"color":["yellow", "green", "purple", "red"], "weight":[124,4.56,384,-2]})
df # shows the dataframe

average = df['weight'].sum() / len(df['weight'])  
average # if you put a variable as the last line, the variable is printed
127.63999999999999

def get_random_sample(dataset: pd.DataFrame, n: int, random_state: int) -> pd.DataFrame:
    return pd.DataFrame()

def get_random_sample(dataset: pd.DataFrame, n: int, random_state: int) -> pd.DataFrame:
    return dataset.sample(n=n,random_state=random_state)

def find_data_type(dataset:pd.DataFrame,column_name:str) -> np.dtype:
    return np.dtype()

def set_index_col(dataset:pd.DataFrame,index:pd.Series) -> pd.DataFrame:
    return pd.DataFrame()

def reset_index_col(dataset:pd.DataFrame) -> pd.DataFrame:
    return pd.DataFrame()

# Set astype (string, int, datetime)
def set_col_type(dataset:pd.DataFrame,column_name:str,new_col_type:type) -> pd.DataFrame:
    return pd.DataFrame()

# Take Matrix of numbers and make it into a DataFrame with column name and index numbering
def make_DF_from_2d_array(array_2d:np.array,column_name_list:list[str],index:pd.Series) -> pd.DataFrame:
    return pd.DataFrame()

# Sort DataFrame by values
def sort_DF_by_column(dataset:pd.DataFrame,column_name:str,descending:bool) -> pd.DataFrame:
    return pd.DataFrame()

# Drop NA values in DataFrame Columns 
def drop_NA_cols(dataset:pd.DataFrame) -> pd.DataFrame:
    return pd.DataFrame()

def drop_NA_rows(dataset:pd.DataFrame) -> pd.DataFrame:
    return pd.DataFrame()

def make_new_column(dataset:pd.DataFrame,new_column_name:str,new_column_value:list) -> pd.DataFrame:
    return pd.DataFrame()

def left_merge_DFs_by_column(left_dataset:pd.DataFrame,right_dataset:pd.DataFrame,join_col_name:str) -> pd.DataFrame:
    return pd.DataFrame()

class simpleClass():
    def __init__(self, length:int, width:int, height:int):
        pass

def find_dataset_statistics(dataset:pd.DataFrame,label_col:str) -> tuple[int,int,int,int,int]:
  n_records = #TODO
  n_columns = #TODO
  n_negative = #TODO
  n_positive = #TODO
  perc_positive = #TODO
  return n_records,n_columns,n_negative,n_positive,perc_positive

Task 2 (25 points)

Now that you have a basic understanding of pandas and the dataset, it is time to dive into some more complex data processing tasks.

Theory

In machine learning a common goal is to train a model on one set of data. Then we validate the model on a similarly structured but different set of data. You could, for example, train the model on data you have collected historically. Then you would validate the model against real-time data as it comes in, seeing how well it predicts the new data coming in.

If you’re looking at a past dataset as we are in these tasks, we need to treat different parts of the data differently to be able to develop and test models. We segregate the data into test and training portions. We train the model on the training data and test the developed model on the test data to see how well it predicts the results.

You should never train your models on test data, only on training data.

Notes

At a high level it is important to hold out a subset of your data when you train a model. You can see what the expected performance is on unseen sample. Thus, you can determine if the resulting model is overfit (performs much better on training data vs test data).

Preprocessing data is essential because most models only take in numerical values. Therefore, categorical features need to be “encoded” to numerical values so that models can use them. A machine learning model may not be able to make sense of “green”, “blue” and “red.” In preprocessing, we’ll convert those to integer values 1, 2 and 3, for example. It’s an interesting question as to what happens when you have training data that has “green,” “red” and blue,” but your testing data says “yellow.”

Numerical scaling can be more or less useful depending on the type of model used, but it is especially important in linear models. Numerical scaling is typically taking positive value and “compressing” them into a range between 0 and 1 (inclusive) that retains the relationships among the original data.

These preprocessing techniques will provide you with options to augment your dataset and improve model performance.

Useful Links:

• Training and Test Sets – Machine Learning – Google Developers
• Bias–variance tradeoff – Wikipedia
• Overfitting – Wikipedia
• Categorical and Numerical Types of Data – 365 Data Science
• scikit-learn: machine learning in Python — scikit-learn 1.2.1 documentation

Deliverables:

• Complete the functions and methods in task2.py
• For this task we have released a local test suite please set that up and use it to debug your function.
• Submit task2.py to Gradescope when you pass all local tests. Refer to the Submissions page for details.

Instructions:

The Task2.py File has function skeletons that you will complete with python code (mostly using the pandas and scikit-learn libraries). The Goal of each of these functions is to give you familiarity with the applied concepts of Splitting and Preprocessing Data. See information about the Function’s Inputs, Outputs and Skeletons below

Local Test Dataset Information

For this task the local test dataset we are using a very simple example dataset, which contains 10 rows and 6 columns of data related to network traffic.

In this task we will guide you through splitting datasets into train and test sets as well as preprocessing datasets using scikit-learn encoders, scalers and dimensionality reduction techniques.

Table of contents

1. tts
2. PreprocessDataset
   1. __init__
   2. One Hot Encoding
   3. Min/Max Scaling
   4. PCA
   5. Feature Engineering
   6. Preprocess

tts

In this function, you will take:

• a dataset
• the name of its label column
• a percentage of the data to put into the test set
• whether you should stratify on the label column
• a random state to set the scikit-learn function

You will return features and labels for the training and test sets.

At a high level, you can separate the task into two subtasks. The first is splitting your dataset into both features and labels (by columns), and the second is splitting your dataset into training and test sets (by rows). You should use the scikit-learn train_test_split function but will have to write wrapper code around it based on the input values we give you.

Useful Resources

• https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html
• https://developers.google.com/machine-learning/crash-course/framing/ml-terminology
• https://stackoverflow.com/questions/40898019/what-is-the-difference-between-a-feature-and-a-label

INPUTS

• dataset – a pandas DataFrame that contains some data
• label_col – a string containing the name of the column that contains the label values (what our model wants to predict)
• test_size – a float containing the decimal value of the percentage of the number of rows that the test set should be out of the dataset
• should_stratify – a boolean (True or False) value indicating if the resulting train/test split should be stratified or not
• random_state – an integer value to set the randomness of the function (useful for repeatability especially when autograding)

OUTPUTS

• train_features – a pandas DataFrame that contains the train rows and the feature columns
• test_features – a pandas DataFrame that contains the test rows and the feature columns
• train_labels – a pandas Series that contains the train rows and the label column
• test_labels – a pandas Series that contains the test rows and the label column

Function Skeleton

def tts(  dataset: pd.DataFrame,
          label_col: str, 
          test_size: float,
          should_stratify: bool,
          random_state: int) -> tuple[pd.DataFrame,pd.DataFrame,pd.Series,pd.Series]:
    # TODO
    return train_features,test_features,train_labels,test_labels

PreprocessDataset

The PreprocessDataset Class contains a code skeleton with nine methods for you to implement. Most methods will be split into two parts: one that will be run on the training dataset and one that will be run on the test dataset. In Data Science/Machine Learning, this is done to avoid something called Data Leakage.

For this assignment, we don’t expect you to understand the nuances of the concept, but we will have you follow principles that will minimize the chances of it occurring. You will accomplish this by splitting data into training and test datasets and processing those datasets in slightly different ways.

Generally, for everything you do in this project, and if you do any ML or Data Science work in the future, you should train/fit on the training data first, then predict/transform on the training and test data. That holds up for basic preprocessing steps like task 2 and for complex models like you will see in tasks 3 and 4.

For the purposes of this project, you should never train or fit on the test data (and more generally in any ML project) because your test data is expected to give you an understanding of how your model/predictions will perform on unseen data. If you fit even a preprocessing step to your test data, then you are either giving the model information about the test set it wouldn’t have about unseen data (if you combine train and test and fit to both), or you are providing a different preprocessing than the model is expecting (if you fit a different preprocessor to the test data), and your model would not be expected to perform well.

Note: You should train/fit using the train dataset; then, once you have a fit encoder/scaler/pca/model instance, you can transform/predict on the training and test data.

You will also notice that we are only preprocessing the Features and not the Labels. There are a few cases where preprocessing steps on labels may be helpful in modeling, but they are definitely more advanced and out of the scope of this introduction. Generally, you will not need to do any preprocessing to your labels beyond potentially encoding a string value (i.e., “Malware” or “Benign”) into an integer value (0 or 1), which is called Label Encoding.

PreprocessDataset:__init__

Similar to the Task1 simpleClass subtask you previously completed you will initialize the class by adding instance variables (add all the inputs to the class).

Useful Resources

• https://www.w3schools.com/python/python_classes.asp

INPUTS

• one_hot_encode_cols – a list of column names (strings) that should be one hot encoded by the one hot encode methods
• min_max_scale_cols – a list of column names (strings) that should be min/max scaled by the min/max scaling methods
• n_components – an int that contains the number of components that should be used in Principal Component Analysis
• feature_engineering_functions – a dictionary that contains feature name and function to create that feature as a key value pair (example shown below)

Example of feature_engineering_functions:

def double_height(dataframe:pd.DataFrame):
    return dataframe["height"] * 2
    
def half_height(dataframe:pd.DataFrame):
    return dataframe["height"] / 2
    
feature_engineering_functions = {"double_height":double_height,"half_height":half_height}

Don’t worry about copying it we also have examples in the local test cases this is just provided as an illustration of what to expect in your function.

OUTPUTS

None, just assign all the input parameters to class variables.

Also per the instructions below, you’ll return here and create another instance variable: a scikit-learn OneHotEncoder with any Parameters you may need later.

Function Skeleton

def __init__(self, 
                 one_hot_encode_cols:list[str],
                 min_max_scale_cols:list[str],
                 n_components:int,
                 feature_engineering_functions:dict
                 ):
        # TODO: Add any instance variables you may need to make your functions work
        return

PreprocessDataset:one_hot_encode_columns_train and one_hot_encode_columns_test

One Hot Encoding is the process of taking a column and returning a binary vector representing the various values within it. There is a separate function for the training and test datasets since they should be handled separately to avoid data leakage (see the 3rd link in Useful Resources for a little more info on how to handle them).

Pseudocode

one_hot_encode_columns_train()

1. In the PreprocessDataset __init__() method initialize an instance variable containing a scikit-learn OneHotEncoder with any parameters you may need.
2. Split train_features into two DataFrames: one with only the columns you want to one hot encode (using one_hot_encode_cols) and another with all the other columns.
3. Fit the OneHotEncoder using the DataFrame you split from train_features with the columns you want to encode.
4. Transform the DataFrame you split from train_features with the columns you want to encode using the fitted OneHotEncoder.
5. Create a DataFrame from the 2D array of data that the output from step 4 gave you, with column names in the form of columnName_categoryName (there should be an attribute in OneHotEncoder that can help you with this) and the same index that train_features had.
6. Concatenate the DataFrame you made in step 5 with the DataFrame of other columns from step 2.

one_hot_encode_columns_test()

1. Split test_features into two DataFrames: one with only the columns you want to one hot encode (usingone_hot_encode_cols) and another with all the other columns.
2. Transform the DataFrame you split from test_features with the columns you want to encode using the OneHotEncoder you fit in one_hot_encode_columns_train()
3. Create a DataFrame from the 2D array of data that the output from step 2 gave you, with column names in the form of columnName_categoryName (there should be an attribute in OneHotEncoder that can help you with this) and the same index that test_features had.
4. Concatenate the DataFrame you made in step 3 with the DataFrame of other columns from step 1.

Example Walkthrough (from Local Testing suite):

INPUTS:

one_hot_encode_cols

["src_ip","protocol"]

Train Features

Test Features

TRAIN DATAFRAMES AT EACH STEP:

2.

DataFrame with columns to encode:

DataFrame with other columns:

4.

One Hot Encoded 2d array:

5.

One Hot Encoded DataFrame with Index and Column Names

6.

Final DataFrame with passthrough/other columns joined back

TEST DATAFRAMES AT EACH STEP:

1.

DataFrame with columns to encode:

DataFrame with other columns:

2.

One Hot Encoded 2d array:

3.

One Hot Encoded DataFrame with Index and Column Names

4.

Final DataFrame with passthrough columns joined back

Note: For the local tests and autograder use the column naming scheme of joining the previous column name and the column value with an underscore (similar to above where Type -> Type_Fruit and Type_Vegetable)

Note 2: Since you should only be fitting your encoder on the training data, if there are values in your test set that are different than those in the training set, you will denote that with 0s. In the example above, let’s say we have a row in the test set with pizza, which is neither a fruit nor vegetable for the Type_Fruit and Type_Vegetable. It should result in a 0 for both columns. If you don’t handle these properly, you may get errors like Test Failed: Found unknown categories.

Note 3: You may be tempted to use the pandas function get_dummies to solve this task, but its a trap. It seems easier, but you will have to do a lot more work to make it handle a train/test split. So, we suggest you use scikit-learn’s OneHotEncoder.

Useful Resources

• https://www.educative.io/blog/one-hot-encoding
• https://developers.google.com/machine-learning/data-prep/transform/transform-categorical
• https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html#sklearn.preprocessing.OneHotEncoder
• https://datascience.stackexchange.com/questions/103211/do-we-need-to-pre-process-both-the-test-and-train-data-set

INPUTS

• Use the needed instance variables you set in the __init__ method
• train_features – a dataset split by a function similar to tts which should be used in the training/fitting steps
• test_features – a dataset split by a function similar to tts which should be used in the test steps

OUTPUTS

a pandas DataFrame with the columns listed in one_hot_encode_cols one hot encoded and all other columns in the DataFrame unchanged

Function Skeleton

def one_hot_encode_columns_train(self,train_features:pd.DataFrame) -> pd.DataFrame:
    one_hot_encoded_dataset = pd.DataFrame()
    return one_hot_encoded_dataset
def one_hot_encode_columns_test(self,test_features:pd.DataFrame) -> pd.DataFrame:
    one_hot_encoded_dataset = pd.DataFrame()
    return one_hot_encoded_dataset

PreprocessDataset:min_max_scaled_columns_train and min_max_scaled_columns_test

Min/Max Scaling is a process to transform numerical features to a specific range, typically [0, 1], to ensure that input values are comparable (similar to how you may have heard of “normalizing” data) and is a crucial preprocessing step for many machine learning algos. In particular this standardization is essential for algorithms like linear regression, logistic regression, k-means, and neural networks, which can be sensitive to the scale of input features, whereas some algos like decision trees are less impacted.

By applying Min/Max Scaling, we prevent feature dominance, to ideally improve performance and accuracy of these algorithms and improve training convergence. It’s a recommended step to ensure your models are trained on consistent and standardized data.

For the provided assignment you should use the scikit-learn MinMaxScaler function (linked in the resources below) rather than attempting to implement your own scaling function.

The rough implementation of the scikit-learn function is provided below for educational purposes.

X_std = (X - X.min(axis=0)) / (X.max(axis=0) - X.min(axis=0))
X_scaled = X_std * (max - min) + min

Note: There are separate functions for the training and test datasets to help avoid data leakage between the test/train datasets. Please refer to the 3rd link in Useful Resources for more information on how to handle this – namely that we should still scale the test data based on our “knowledge” of the train dataset.

Example Dataframe:

Example Min Max Scaled Dataframe (rounded to 4 decimal places):

Note: For the Autograder use the same column name as the original column (ex: Price -> Price)

Useful Resources

• https://developers.google.com/machine-learning/data-prep/transform/normalization
• https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html#sklearn.preprocessing.MinMaxScaler
• https://datascience.stackexchange.com/questions/103211/do-we-need-to-pre-process-both-the-test-and-train-data-set
• https://www.geeksforgeeks.org/pandas/python-scaling-numbers-column-by-column-with-pandas/

INPUTS

• Use the needed instance variables you set in the __init__ method
• train_features – a dataset split by a function similar to tts which should be used in the training/fitting steps
• test_features – a dataset split by a function similar to tts which should be used in the test steps

OUTPUTS

a pandas DataFrame with the columns listed in min_max_scale_cols min/max scaled and all other columns in the DataFrame unchanged

Function Skeleton

    def min_max_scaled_columns_train(self,train_features:pd.DataFrame) -> pd.DataFrame:
        min_max_scaled_dataset = pd.DataFrame()
        return min_max_scaled_dataset
    def min_max_scaled_columns_test(self,test_features:pd.DataFrame) -> pd.DataFrame:
        min_max_scaled_dataset = pd.DataFrame()
        return min_max_scaled_dataset

PreprocessDataset:pca_train and pca_test

Principal Component Analysis is a dimensionality reduction technique (column reduction). It aims to take the variance in your input columns and map the columns into N columns that contain as much of the variance as it can. This technique can be useful if you are trying to train a model faster and has some more advanced uses, especially when training models on data which has many columns but few rows. There is a separate function for the training and test datasets because they should be handled separately to avoid data leakage (see the 3rd link in Useful Resources for a little more info on how to handle them).

Note 1: For the local tests and autograder, use the following column naming scheme: component_1, component_2, …, component_n, where n is the n_components value passed to the __init__ method.

Note 2: For your PCA outputs to match the local tests and autograder, make sure you set the seed using a random state of 0 when you initialize the PCA function.

Note 3: Since PCA does not work with NA values, make sure you drop any columns that have NA values before running PCA.

Useful Resources

• https://builtin.com/data-science/step-step-explanation-principal-component-analysis
• https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html#sklearn.decomposition.PCA
• https://datascience.stackexchange.com/questions/103211/do-we-need-to-pre-process-both-the-test-and-train-data-set

INPUTS

• Use the needed instance variables you set in the __init__ method
• train_features – a dataset split by a function similar to tts which should be used in the training/fitting steps
• test_features – a dataset split by a function similar to tts which should be used in the test steps

OUTPUTS

A pandas DataFrame with the generated PCA values and the column names component_1, component_2, …, component_n.

Function Skeleton

 def pca_train(self,train_features:pd.DataFrame) -> pd.DataFrame:
     # TODO: Read the function description in https://github.gatech.edu/pages/cs6035-tools/cs6035-tools.github.io/Projects/Machine_Learning/Task2.html and implement the function as described
     pca_dataset = pd.DataFrame()
     return pca_dataset

 def pca_test(self,test_features:pd.DataFrame) -> pd.DataFrame:
     # TODO: Read the function description in https://github.gatech.edu/pages/cs6035-tools/cs6035-tools.github.io/Projects/Machine_Learning/Task2.html and implement the function as described
     pca_dataset = pd.DataFrame()
     return pca_dataset

PreprocessDataset:feature_engineering_train, feature_engineering_test

Feature Engineering is a process of using domain knowledge (physics, geometry, sports statistics, business metrics, etc.) to create new features (columns) out of the existing data. This could mean creating an area feature when given the length and width of a triangle or extracting the major and minor version number from a software version or more complex logic depending on the scenario.

In cybersecurity in particular, feature engineering is crucial for using domain expert’s (e.g. a security analyst) experience to identify anomalous behavior that might signify a security breach. This could involve creating features that represent deviations from established baselines, such as unusual file access patterns, unexpected network connections, or sudden spikes in CPU usage. These anomaly-based features can help distinguish malicious activity from normal system operations, but the system does not know what data patterns mean anomalous off-hand – that is where you as the domain expert can help by creating features.

These methods utilize a dictionary, feature_engineering_functions, passed to the class constructor (__init__). This dictionary defines how to generate new features:

1. Keys: Strings representing new column names.
2. Values: Functions that:
   • Take a DataFrame as input.
   • Return a Pandas Series (the new column’s values).

Example of what could be passed as the feature_engineering_functions dictionary to __init__:

import pandas as pd
def double_height(dataframe: pd.DataFrame) -> pd.Series:
    return dataframe["height"] * 2
    
def half_height(dataframe: pd.DataFrame) -> pd.Series:
    return dataframe["height"] / 2
    
example_feature_engineering_functions = {
    "double_height": double_height, # Note that functions in python can be passed around and used just like data!
    "half_height": half_height
}

# and the class may be been created like this...
# preprocessor = PreprocessDataset(..., feature_engineering_functions=example_feature_engineering_functions, ...)

In particular for this method, you will be taking in a dictionary with a column name and a function that takes in a DataFrame and returns a column. You’ll be using that to create a new column with the name in the dictionary key. Therefore, if you were given the above functions, you would create two new columns named “double_height” and “half_height” in your DataFrame.

Useful Resources

• https://en.wikipedia.org/wiki/Feature_engineering
• https://www.geeksforgeeks.org/what-is-feature-engineering/
• Passing Function as an Argument in Python – GeeksforGeeks

INPUTS

• Use the needed instance variables you set in the __init__ method
• train_features – a dataset split by a function similar to tts which should be used in the training/fitting steps
• test_features – a dataset split by a function similar to tts which should be used in the test steps

OUTPUTS

a pandas dataframe with the features described in feature_engineering_train and feature_engineering_test added as new columns and all other columns in the dataframe unchanged

Function Skeleton

def feature_engineering_train(self,train_features:pd.DataFrame) -> pd.DataFrame:
    feature_engineered_dataset = pd.DataFrame()
    return feature_engineered_dataset
def feature_engineering_test(self,test_features:pd.DataFrame) -> pd.DataFrame:
    feature_engineered_dataset = pd.DataFrame()
    return feature_engineered_dataset

PreprocessDataset:preprocess_train, preprocess_test

Now, we will put three of the above methods together into a preprocess function. This function will take in a dataset and perform encoding, scaling, and feature engineering using the above methods and their respective columns. You should not perform PCA for this function.

Useful Resources

See resources for one hot encoding, min/max scaling and feature engineering above

INPUTS

• Use the needed instance variables you set in the __init__ method
• train_features – a dataset split by a function similar to tts which should be used in the training/fitting steps
• test_features – a dataset split by a function similar to tts which should be used in the test steps

OUTPUTS

A pandas DataFrame for both test and train features with the columns in one_hot_encode_cols encoded, the columns in min_max_scale_cols scaled, and the columns described in feature_engineering_functions engineered. You do not need to use PCA here.

Function Skeleton

def preprocess_train(self,train_features:pd.DataFrame) -> pd.DataFrame:
  preprocessed_dataset = pd.DataFrame()
  return train_features

def preprocess_test(self,test_features:pd.DataFrame) -> pd.DataFrame:
  preprocessed_dataset = pd.DataFrame()

return test_features

Task 3 (15 points)

In Task 2 you learned how to split a dataset into training and testing components. Now it’s time to learn about using a K-means model. We will run a basic model on the data to cluster files (rows) with similar attributes together. We will use an unsupervised model.

Theory

An unsupervised model has no label column. By contrast, in supervised learning (which you’ll see in Task 4) the data has features and targets/labels. These labels are effectively an answer key to the data in the feature columns. You don’t have this answer key in unsupervised learning, instead you’re working on data without labels. You’ll need to use algorithms that learn directly from the data without the benefit of labels.

We start with K-means because it is simple to understand the algorithm. For the mathematical people, you can look at the underlying data structure, a Voronoi diagram. Based on squared Euclidean distances, K-means creates clusters of similar datapoints. Each cluster has a centroid. The idea is that for each sample, it’s associated/clustered with the centroid that is the “closest.”

Closest is an interesting concept in higher dimensions. You can think of each feature in a dataset as a dimension in the data. If it’s 2D or 3D, we can visualize it easily. Concepts of distance are easy to visualize in 2D and 3D, and they extend similarly to higher dimensions.

If you read the Wikipedia articles for K-means you’ll see a discussion of the use of “squared Euclidean distances” in K-means. This is compared with simple Euclidean distances in the Weber problem, and better approaches resulting from k-medians and k-mediods is discussed.

Please use scikit-learn to create the model and Yellowbrick to determine the optimal value of k for the dataset.

So far, we have functions to split the data and preprocess it. Now, we will run a basic model on the data to cluster files (rows) with similar attributes together. We will use an unsupervised model (model with no label column), K-means. Again, use scikit-learn to create the model and Yellowbrick to determine the optimal value of k for the dataset.

Refer to the Submissions page for details about submitting your work.

Useful Links:

• Clustering – Google Developers
• Clustering Algorithms – Google Developers
• Kmeans – Google Developers

Deliverables:

• Complete the KmeansClustering class in task3.py.
• For this task we have released a local test suite please set that up and use it to debug your function.
• Submit task3.py to Gradescope when you pass all local tests. Refer to the Submissions page for details.
  

  Local Test Dataset Information

  For this task the local test dataset we are using is the NATICUSdroid dataset, which contains 86 columns of data related to android permissions used by benign and malicious Android applications released between 2010 and 2019. For more information such as the introductory paper and the Citations/Acknowledgements you can view the dataset site in the UCI ML repository. In this specific case clustering can be a useful tool to group apps that request similar permissions together. The team that created this dataset hypothesized that malicious apps would exhibit distinct patterns in the types of permissions they request compared to benign apps. This difference in permission request patterns could potentially be used to distinguish between malicious and benign applications.

  KmeansClustering

  The KmeansClustering Class contains a code skeleton with 4 methods for you to implement.

Note: You should train/fit using the train dataset then once you have a Yellowbrick/K-means model instance you can transform/predict on the training and test data.

KmeansClustering:__init__

Similar to Task 1, you will initialize the class by adding instance variables as needed.

Useful Resources

• https://www.w3schools.com/python/python_classes.asp

INPUTS

These inputs are all expected to be passed to the sklearn.cluster.KMeans class so review that documentation for more details on what the parameters are used for.

• random_state – an integer that should be used to set the scikit-learn randomness so the model results will be repeatable which is required for the tests and autograder
• init – a string denoting what method to use to initialize kmeans cluster centroids so the model results will be repeatable which is required for the tests and autograder
• n_init – an integer used to set the number of times the k-means algorithm is run with different centroid seeds so the model results will be repeatable which is required for the tests and autograder
• max_iter – an integer used to set the maximum number of iterations of the k-means algorithm for a single run so the model results will be repeatable which is required for the tests and autograder
• algorithm – a string denoting what kmeans algorithm to use so the model results will be repeatable which is required for the tests and autograder
• tol – a float used to set the threshold for relative tolerance for the difference in the cluster centers of two consecutive iterations to declare convergence so the model results will be repeatable which is required for the tests and autograder

OUTPUTS

None

Function Skeleton

def __init__(self, 
             random_state: int,
             init: str, 
             n_init: int,
             max_iter: int,
             algorithm: str,
             tol: float
            ):
  # TODO: Add any state variables you may need to make your functions work
  pass

KmeansClustering:kmeans_get_n_clusters

K-means clustering is a process of grouping similar rows together and assigning them to a cluster. For this method, you will use the training data to fit an optimal K-means cluster on the data.

To help you get started we have provided a list of subtasks to complete for this task:

1. Initialize a scikit-learn K-means model with the parameters passed to the __init__ method
   • init
   • n_init
   • max_iter
   • random_state
   • algorithm
   • tol
2. Try to find the best k to use for K-means clustering.
   • Initialize a Yellowbrick KElbowVisualizer with the K-means model.
   • Use that visualizer to search for the optimal value of k (between 2 inclusive and 10 exclusive; in mathematical notation, [2,10)).
   • If you are stuck on this step, review the Yellowbrick docs.
3. Train the KElbowVisualizer on the training data and determine the optimal k value.

Useful Resources

• https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html#sklearn.cluster.KMeans
• https://www.scikit-yb.org/en/latest/api/cluster/elbow.html

INPUTS

• Use the needed instance variables you set in the __init__ method
• train_features – a dataset split by a function similar to tts which should be used in the training/fitting steps

OUTPUTS

The optimal value of K for the input training features

Function Skeleton

def kmeans_get_n_clusters(self, train_features:pd.DataFrame) -> int:
    k = int
    return k

KmeansClustering:kmeans_train

K-means clustering is a process of grouping similar rows together and assigning them to a cluster. For this method, you will use the training data to fit an optimal K-means cluster on the data.

1. Train a K-means model with the proper initialization and using either the value of k passed as an input or if no value is passed determine the optimal value of k (using the function you previously wrote kmeans_get_n_clusters)
2. Return the cluster ids for each row of the training set as a list.

Useful Resources

• https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html#sklearn.cluster.KMeans

INPUTS

• Use the needed instance variables you set in the __init__ method
• train_features – a dataset split by a function similar to tts which should be used in the training/fitting steps
• k (optional) – an int value of the number of kmeans clusters or None

OUTPUTS

a list of cluster ids that the K-means model has assigned for each row in the train dataset

Function Skeleton

def kmeans_train(self, train_features:pd.DataFrame, k:int=None) -> list:
    cluster_ids = list()
    return cluster_ids

KmeansClustering:kmeans_test

K-means clustering is a process of grouping together similar rows together and assigning them to a cluster. For this method you will use the trained model from the previous task to fit an optimal K-means cluster on the test data.

To help you get started, we have provided a list of subtasks to complete for this task:

1. Use the model you trained in the kmeans_train method to generate cluster ids for each row of the test dataset.
   • If you are stuck here review the scikit-learn docs for KMeans and make sure you are using a method that will return cluster ids for the test data you input.
2. Return the cluster ids for each row of the test set as a list.

Useful Resources

• https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html#sklearn.cluster.KMeans

INPUTS

• Use the needed instance variables you set in the __init__ method
• test_features – a dataset split by a function similar to tts which should be used in the test steps

OUTPUTS

a list of cluster ids that the K-means model has assigned for each row in the test dataset

Function Skeleton

def kmeans_test(self, test_features:pd.DataFrame) -> list:
        cluster_ids = list()
        return cluster_ids

Task 4 (25 points)

Now let’s try a few supervised classification models, we have chosen a few commonly used models for you to use here, but there are many options. In the real world, specific algorithms may fit a specific dataset better than other algorithms.

You won’t be doing any hyperparameter tuning yet, so you can better focus on writing the basic code. You will:

• Train a model using the training set.
• Predict on the training/test sets.
• Calculate performance metrics.
• Return a ModelMetrics object and trained scikit-learn model from each model function.

Important Note on Feature Selection: You should ONLY use RFE (Recursive Feature Elimination) for determining feature importance of your Logistic Regression model. Do NOT use RFE for any tree-based models (Decision Tree, Random Forest, or Gradient Boosting). For tree-based models, use their built-in feature importance values instead.

Useful Links:

• scikit-learn: machine learning in Python — scikit-learn 1.2.1 documentation
• Training and Test Sets – Machine Learning – Google Developers
• Bias–variance tradeoff – Wikipedia
• Overfitting – Wikipedia
• An Introduction to Classification in Machine Learning – builtin
• Classification in Machine Learning: An Introduction – DataCamp

Deliverables:

• Complete the functions and methods in task4.py
• For this task we have released a local test suite please set that up and use it to debug your function.
• Submit task4.py to Gradescope when you pass all local tests. Refer to the Submissions page for details.
  

  Local Test Dataset Information

  For this task the local test dataset we are using is the NATICUSdroid dataset, which contains 86 columns of data related to android permissions used by benign and malicious Android applications released between 2010 and 2019. For more information such as the introductory paper and the Citations/Acknowledgements you can view the dataset site in the UCI ML repository. If you look at the online poster for the paper that the dataset creators wrote from their research, they trained a variety of different models including Random Forest, Logistic Regression and XGBoost and calculated a variety of metrics related to training and detection performance. In this task we will guide you through training ML models and calculating performance metrics to compare the predictive abilities of different models.

Instructions:

The Task4.py File has function skeletons that you will complete with Python code (mostly using the pandas and scikit-learn libraries).

The goal of each of these functions is to give you familiarity with the applied concepts of training a model, using it to score records and calculating performance metrics for it. See information about the function inputs, outputs and skeletons below.

Table of contents

1. ModelMetrics
2. calculate_naive_metrics
3. calculate_logistic_regression_metrics
4. calculate_decision_tree_metrics
5. calculate_gradient_boosting_metrics

ModelMetrics

• In order to simplify the autograding we have created a class that will hold the metrics and feature importances for a model you trained.
• You should not modify this class but are expected to use it in your return statements.
• This means you put your training and test metrics dictionaries and feature importance DataFrames inside a ModelMetrics object for the autograder to handle. This is for each of the Logistic Regression, Gradient Boosting and Random Forest models you will create.
• You do not need to return a feature importance DataFrame in the ModelMetrics value for the naive model you will create, just return None in that position of the return statement, as the given code does.

calculate_naive_metrics

A Naive model is a very simple model/prediction that can help to frame how well a more sophisticated model is doing. At best, such a model has random competence at predicting things. At worst, it’s wrong all the time.

Since a naive model is incredibly basic (often a constant or randomly selected result), we can expect that any more sophisticated model that we train should outperform it. If the naive Model beats our trained model, it can mean that additional data (rows or columns) is needed in the dataset to improve our model. It can also mean that the dataset doesn’t have a strong enough signal for the target we want to predict.

In this function, you’ll implement a simple model that always predicts a constant (function-provided) number, regardless of the input values. Specifically, you’ll use a given constant integer, provided as the parameter naive_assumption, as the model’s prediction. This means the model will always output this constant value, without considering the actual data. Afterward, you will calculate four metrics—accuracy, recall, precision, and F1-score—for both the training and test datasets.

[1] Refer to the resources below.

Useful Resources

• https://machinelearningmastery.com/how-to-develop-and-evaluate-naive-classifier-strategies-using-probability/
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score

INPUTS

• train_features – a dataset split by a function similar to the tts function you created in task2
• test_features – a dataset split by a function similar to the tts function you created in task2
• train_targets – a dataset split by a function similar to the tts function you created in task2
• test_targets – a dataset split by a function similar to the tts function you created in task2
• naive_assumption – an integer that should be used as the result from the naive model you will create

OUTPUTS

A completed ModelMetrics object with a training and test metrics dictionary with each one of the metrics rounded to 4 decimal places

Function Skeleton

def calculate_naive_metrics(train_features:pd.DataFrame, test_features:pd.DataFrame, train_targets:pd.Series, test_targets:pd.Series, naive_assumption:int) -> ModelMetrics:
    train_metrics = {
        "accuracy" : 0,
        "recall" : 0,
        "precision" : 0,
        "fscore" : 0
        }
    test_metrics = {
        "accuracy" : 0,
        "recall" : 0,
        "precision" : 0,
        "fscore" : 0
        }
    naive_metrics = ModelMetrics("Naive",train_metrics,test_metrics,None)
    return naive_metrics

calculate_logistic_regression_metrics

Important Note on Feature Selection: RFE (Recursive Feature Elimination) should ONLY be used for Logistic Regression. Do NOT use RFE for any tree-based models (Decision Tree, Random Forest, or Gradient Boosting) – use their built-in feature importance values instead.

A logistic regression model is a simple and more explainable statistical model that can be used to estimate the probability of an event (log-odds). At a high level, a logistic regression model uses data in the training set to estimate a column’s weight in a linear approximation function. Conceptually this is similar to estimating m for each column in the line formula you probably know well from geometry: y = m*x + b. If you are interested in learning more, you can read up on the math behind how this works. For this project, we are more focused on showing you how to apply these models, so you can simply use a scikit-learn Logistic Regression model in your code.

For this task use scikit-learn’s LogisticRegression class and complete the following subtasks:

• Train a Logistic Regression model (initialized using the kwargs passed into the function)
• Predict scores for training and test datasets and calculate the 7 metrics listed below for the training and test datasets using predictions from the fit model. (All rounded to 4 decimal places)
  • accuracy
  • recall
  • precision
  • fscore
  • false positive rate (fpr)
  • false negative rate (fnr)
  • Area Under the Curve of Receiver Operating Characteristics Curve (roc_auc)
• Use RFE to select the top n_feat_importance features
• Train a Logistic Regression model using these selected features (initialized using the kwargs passed into the function)
• Create a Feature Importance DataFrame from the model trained on the top n_feat_importance features:
  • Use the top N features (where N is configured via n_feat_importance input variable) and sort by absolute value of the coefficient from biggest to smallest.
  • Make sure you use the same feature and importance column names as set in ModelMetrics in feat_name_col [Feature] and imp_col [Importance].
  • Round the importances to 4 decimal places (do this step after you have sorted by Importance)
  • Reset the index to 0->(N-1). You can do this the same way you did in task1.

NOTE: Make sure you use the predicted probabilities for roc_auc.

Useful Resources

• https://stats.libretexts.org/Bookshelves/Introductory_Statistics/OpenIntro_Statistics_(Diez_et_al)./08%3A_Multiple_and_Logistic_Regression/8.04%3A_Introduction_to_Logistic_Regression
• https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score
• https://en.wikipedia.org/wiki/Confusion_matrix
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html

INPUTS

The first 4 are similar to the tts function you created in Task 2:

• train_features – a Pandas DataFrame with training features
• test_features – a Pandas DataFrame with test features
• train_targets – a Pandas DataFrame with training targets
• test_targets – a Pandas DataFrame with test targets
• n_feat_importance – an int with how many features to return in the feature importance dataframe
• logreg_kwargs – a dictionary with keyword arguments that can be passed directly to the scikit-learn Logistic Regression class

OUTPUTS

• A completed ModelMetrics object with a training and test metrics dictionary with each one of the metrics rounded to 4 decimal places
• A scikit-learn Logistic Regression model object fit on the training set

Function Skeleton

def calculate_logistic_regression_metrics(train_features:pd.DataFrame, test_features:pd.DataFrame, train_targets:pd.Series, test_targets:pd.Series, n_feat_importance: int, logreg_kwargs) -> tuple[ModelMetrics,LogisticRegression]:
    model = LogisticRegression()
    train_metrics = {
        "accuracy" : 0,
        "recall" : 0,
        "precision" : 0,
        "fscore" : 0,
        "fpr" : 0,
        "fnr" : 0,
        "roc_auc" : 0
        }
    test_metrics = {
        "accuracy" : 0,
        "recall" : 0,
        "precision" : 0,
        "fscore" : 0,
        "fpr" : 0,
        "fnr" : 0,
        "roc_auc" : 0
        }

    log_reg_importance = pd.DataFrame()
    log_reg_metrics = ModelMetrics("Logistic Regression",train_metrics,test_metrics,log_reg_importance)

    return log_reg_metrics,model

Example of Feature Importance DataFrame (with n_feat_importance=10)

calculate_decision_tree_metrics

Important Note on Feature Selection: RFE (Recursive Feature Elimination) should ONLY be used for Logistic Regression. Do NOT use RFE for any tree-based models (Decision Tree, Random Forest, or Gradient Boosting). For tree-based models, use their built-in feature importance values instead.

A Decision Tree (DT) is a supervised learning algorithm used for both classification and regression tasks. It works by recursively splitting the data into subsets based on the feature that results in the best separation of classes, typically measured using Gini impurity or entropy. Decision trees are interpretable, as the learned model can be visualized as a flowchart-like structure.

If you are interested in learning more, you can read up on the math behind how decision trees work.

For this project, we are more focused on showing you how to apply these models, so you can simply use a scikit-learn DecisionTreeClassifier in your code.

For this task, use scikit-learn’s DecisionTreeClassifier class and complete the following subtasks:

• Train a DT model (initialized using the kwargs passed into the function).
• Predict scores for training and test datasets and calculate the 7 metrics listed below for the training and test datasets using predictions from the fit model. (All rounded to 4 decimal places)
  • accuracy
  • recall
  • precision
  • fscore
  • false positive rate (fpr)
  • false negative rate (fnr)
  • Area Under the Curve of Receiver Operating Characteristics Curve (roc_auc)
• Create a Feature Importance DataFrame from the trained model:
  • Do Not Use RFE for feature selection
  • Use the top N (where N is configured via n_feat_importance input variable) features selected by the built in method (sorted from biggest to smallest)
  • Make sure you use the same feature and importance column names as ModelMetrics shows in feat_name_col [Feature] and imp_col [Importance]
  • Round the importances to 4 decimal places (do this step after you have sorted by Importance)
  • Reset the index to 0->(N-1) you can do this the same way you did in task1

NOTE: Make sure you use the predicted probabilities for roc_auc.

Useful Resources

• https://scikit-learn.org/stable/modules/tree.html
• https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score
• https://en.wikipedia.org/wiki/Confusion_matrix
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html

INPUTS

The first 4 are similar to the tts function you created in Task 2:

• train_features – a Pandas DataFrame with training features
• test_features – a Pandas DataFrame with test features
• train_targets – a Pandas DataFrame with training targets
• test_targets – a Pandas DataFrame with test targets
• n_feat_importance – an int with how many features to return in the feature importance dataframe
• dt_kwargs – a dictionary with keyword arguments that can be passed directly to the scikit-learn DecisionTreeClassifier class

OUTPUTS

• A completed ModelMetrics object with a training and test metrics dictionary with each one of the metrics rounded to 4 decimal places
• A scikit-learn DecisionTreeClassifier model object fit on the training set

Function Skeleton

def calculate_decision_tree_metrics(train_features: pd.DataFrame, test_features: pd.DataFrame, train_targets: pd.Series, test_targets: pd.Series, n_feat_importance: int, dt_kwargs) -> tuple[ModelMetrics, DecisionTreeClassifier]:
    model = DecisionTreeClassifier(**dt_kwargs)
    train_metrics = {
        "accuracy": 0,
        "recall": 0,
        "precision": 0,
        "fscore": 0,
        "fpr": 0,
        "fnr": 0,
        "roc_auc": 0
    }
    test_metrics = {
        "accuracy": 0,
        "recall": 0,
        "precision": 0,
        "fscore": 0,
        "fpr": 0,
        "fnr": 0,
        "roc_auc": 0
    }

    dt_importance = pd.DataFrame()
    dt_metrics = ModelMetrics("Decision Tree", train_metrics, test_metrics, dt_importance)

    return dt_metrics, model

Example of Feature Importance DataFrame (with n_feat_importance=5)

calculate_gradient_boosting_metrics

Important Note on Feature Selection: RFE (Recursive Feature Elimination) should ONLY be used for Logistic Regression. Do NOT use RFE for any tree-based models (Decision Tree, Random Forest, or Gradient Boosting). For tree-based models, use their built-in feature importance values instead.

A Gradient Boosted model is more complex than the Naive and Logistic Regression models and similar in structure to a Decision Tree model. A Gradient Boosted model expands on the tree-based model by using its additional trees to predict the errors from the previous tree. For this project, we are more focused on showing you how to apply these models, so you can simply use the scikit-learn Gradient Boosted Model in your code.

For this task use scikit-learn’s Gradient Boosting Classifier class and complete the following subtasks:

• Train a Gradient Boosted model (initialized using the kwargs passed into the function)
• Predict scores for training and test datasets and calculate the 7 metrics listed below for the training and test datasets using predictions from the fit model. (All rounded to 4 decimal places)
  • accuracy
  • recall
  • precision
  • fscore
  • false positive rate (fpr)
  • false negative rate (fnr)
  • Area Under the Curve of Receiver Operating Characteristics Curve (roc_auc)
• Create a Feature Importance DataFrame from the trained model:
  • Do Not Use RFE for feature selection
  • Use the top N (where N is configured via n_feat_importance input variable) features selected by the built in method (sorted from biggest to smallest)
  • Make sure you use the same feature and importance column names as ModelMetrics shows in feat_name_col [Feature] and imp_col [Importance]
  • Round the importances to 4 decimal places (do this step after you have sorted by Importance)
  • Reset the index to 0->(N-1). You can do this the same way you did in Task 1.

NOTE: Make sure you use the predicted probabilities for roc_auc.

Refer to the Submissions page for details about submitting your work.

Useful Resources

• https://blog.dataiku.com/tree-based-models-how-they-work-in-plain-english
• https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score
• https://en.wikipedia.org/wiki/Confusion_matrix
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html

INPUTS

• train_features – a dataset split by a function similar to the tts function you created in task2
• test_features – a dataset split by a function similar to the tts function you created in task2
• train_targets – a dataset split by a function similar to the tts function you created in task2
• test_targets – a dataset split by a function similar to the tts function you created in task2
• n_feat_importance – an int with how many features to return in the feature importance dataframe
• gb_kwargs – a dictionary with keyword arguments that can be passed directly to the scikit-learn GradientBoostingClassifier class

OUTPUTS

• A completed ModelMetrics object with a training and test metrics dictionary with each one of the metrics rounded to 4 decimal places
• An scikit-learn Gradient Boosted model object fit on the training set

Function Skeleton

def calculate_gradient_boosting_metrics(train_features:pd.DataFrame, test_features:pd.DataFrame, train_targets:pd.Series, test_targets:pd.Series, n_feat_importance: int, gb_kwargs) -> tuple[ModelMetrics,GradientBoostingClassifier]:
    model = GradientBoostingClassifier()
    train_metrics = {
        "accuracy" : 0,
        "recall" : 0,
        "precision" : 0,
        "fscore" : 0,
        "fpr" : 0,
        "fnr" : 0,
        "roc_auc" : 0
        }
    test_metrics = {
        "accuracy" : 0,
        "recall" : 0,
        "precision" : 0,
        "fscore" : 0,
        "fpr" : 0,
        "fnr" : 0,
        "roc_auc" : 0
        }

    gb_importance = pd.DataFrame()
    gb_metrics = ModelMetrics("Gradient Boosting",train_metrics,test_metrics,gb_importance)

    return gb_metrics,model

Example of Feature Importance DataFrame (with n_feat_importance=10)

import pandas as pd

def document_hyperparameter_tuning_clamp(train_df,test_df):
    """
    Please document the hyperparameter tuning process you used to tune your machine learning model for Task 5.

    You should copy and paste the hyperparameter process you conducted here.

    Place all parameters tuned and values in the hyperparameters dictionary.

    If we run your code and your hyperparameter function, it must generate the same hyperparameters you used for your function.
    
    You do not need to return anything specific, just document how you did your tuning.

    We will not run it in the autograder, it is just an additional check on our end.
    """

    hyperparameters = {}

    return hyperparameters

def train_model_return_scores_clamp(train_df, test_df) -> pd.DataFrame:
    """
    Trains a model on the ClaMP training data and returns predicted probabilities 
    for the test data.

    Args:
        train_df (pd.DataFrame): ClaMP training data with 'label' column.
        test_df (pd.DataFrame): ClaMP test data without 'label' column.

    Returns:
        pd.DataFrame: DataFrame with 'index' and 'prob_label_1' columns.
    """
    # TODO: Implement the model training and prediction logic as described above.
    test_scores = pd.DataFrame()  # Replace with your implementation
    return test_scores

import pandas as pd

def document_hyperparameter_tuning_unsw(train_df,test_df):
    """
    Please document the hyperparameter tuning process you used to tune your machine learning model for Task 5.

    You should copy and paste the hyperparameter process you conducted here.

    Place all parameters tuned and values in the hyperparameters dictionary.
    
    If we run your code and your hyperparameter function, it must generate the same hyperparameters you used for your function.
    
    You do not need to return anything specific, just document how you did your tuning.

    We will not run it in the autograder, it is just an additional check on our end.
    """

    hyperparameters = {}

    return hyperparameters

def train_model_return_scores_unsw(train_df, test_df) -> pd.DataFrame:
    """
    Trains a model on the UNSW-NB15 training data and returns predicted 
    probabilities for the test data.

    Args:
        train_df (pd.DataFrame): UNSW-NB15 training data with 'label' column.
        test_df (pd.DataFrame): UNSW-NB15 test data without 'label' column.

    Returns:
        pd.DataFrame: DataFrame with 'index' and 'prob_label_1' columns.
    """
    # TODO: Implement the model training and prediction logic as described above.
    test_scores = pd.DataFrame()  # Replace with your implementation
    return test_scores

import pandas as pd

def document_hyperparameter_tuning_phiusiil(train_df,test_df):
    """
    Please document the hyperparameter tuning process you used to tune your machine learning model for Task 5.
    You should copy and paste the hyperparameter process you conducted here.
    Place all parameters tuned and values in the hyperparameters dictionary.
    If we run your code and your hyperparameter function, it must generate the same hyperparameters you used for your function.
    You do not need to return anything specific, just document how you did your tuning.
    We will not run it in the autograder, it is just an additional check on our end.
    """

    hyperparameters = {}

    return hyperparameters

def train_model_return_scores_phiusiil(train_df, test_df) -> pd.DataFrame:
    """
    Trains a model on the PhiUSIIL Phishing URL training data and returns predicted 
    probabilities for the test data.

    Args:
        train_df (pd.DataFrame): PhiUSIIL Phishing URL training data with 'label' column.
        test_df (pd.DataFrame): PhiUSIIL Phishing URL test data without 'label' column.

    Returns:
        pd.DataFrame: DataFrame with 'index' and 'prob_label_1' columns.
    """
    # TODO: Implement the model training and prediction logic as described above.
    test_scores = pd.DataFrame()  # Replace with your implementation
    return test_scores