[SOLVED] EE5934 Project 1

For this project, you will use a pre-trained deep neural network, SqueezeNet, which is lightweight and runs fast on CPUs. Run the code below to load a pre-trained SqueezeNet from the PyTorch official model zoo.

In [2]:

# Test and set the device.if torch.cuda.is_available():    device = ‘cuda:0’else:    device = ‘cpu’print(‘Use’, device) # Download and load the pretrained SqueezeNet model.model = torchvision.models.squeezenet1_1(pretrained=True).to(device) # Disable the gradient computation with respect to model parameters.for param in model.parameters():    param.requires_grad = False

Use cuda:0

Data¶

For Task#1 and Task#2, use the images in folder Project1\images where the filenames are the corresponding class labels. For example, 182.png is an image of Border Terrier, which is class 182 in ImageNet dataset. Please refer to this Gist snippet for a complete list. The images are from ImageNet validation set, and so the pre-trained model has never “seen” them.

For Task#3, you may use the images in folder Project1\style or any other images you like.

Helper Functions¶

Most pre-trained models are trained on images that had been preprocessed by subtracting the per-color mean and dividing by the per-color standard deviation. Here are a few helper functions for performing and undoing this preprocessing.

In [3]:

IMAGENET_MEAN = np.array([0.485, 0.456, 0.406])IMAGENET_STD = np.array([0.229, 0.224, 0.225]) def preprocess(img, size=(224, 224)):    transform = T.Compose([        T.Resize(size),        T.ToTensor(),        T.Normalize(mean=IMAGENET_MEAN.tolist(),                    std=IMAGENET_STD.tolist()),        T.Lambda(lambda x: x[None]),    ])    return transform(img) def deprocess(img, should_rescale=True):    transform = T.Compose([        T.Lambda(lambda x: x[0]),        T.Normalize(mean=[0, 0, 0], std=(1.0 / IMAGENET_STD).tolist()),        T.Normalize(mean=(-IMAGENET_MEAN).tolist(), std=[1, 1, 1]),        T.Lambda(rescale) if should_rescale else T.Lambda(lambda x: x),        T.ToPILImage(),    ])    return transform(img) def rescale(x):    low, high = x.min(), x.max()    x_rescaled = (x – low) / (high – low)    return x_rescaled def blur_image(X, sigma=1):    X_np = X.cpu().clone().numpy()    X_np = gaussian_filter1d(X_np, sigma, axis=2)    X_np = gaussian_filter1d(X_np, sigma, axis=3)    X.copy_(torch.Tensor(X_np).type_as(X))    return X

Task#1 Adversarial Attack¶

The concept of “image gradients” can be used to study the stability of a network. Consider a state-of-the-art deep neural network that generalizes well on an object recognition task. We expect such network to be robust to small perturbations to its input, because small perturbations cannot change the object category of an image. However, it was shown in the following paper[1] that by applying an imperceptible non-random perturbation to a test image, it is possible to arbitrarily change the network’s prediction.

[1] Szegedy et al, “Intriguing properties of neural networks”, ICLR 2014

Given an image and a target class, we can perform gradient ascent over the image to maximize the target class, stopping when the network classifies the image as the target class. While the perturbations seem negligible to humans, the network would classify the perturbed images wrongly.

Read the paper, and then implement the following function make_adversarial_attack to generate “fooling images”. For each image in Project1/images with class label $c$, generate a fooling image that will be classified into class $ c-1-d $ where $d$ is the last digit of your student number. Save each fooling image into the folder Project1/fooling_images with the filename {true_class}_{target_class}.png. You may confirm (optional) that the fooling image 182_9.png in the folder will be wrongly classified as ostrich (class 9 in ImageNet dataset).

For the image 182.png, show the difference map between the original image and the fooling image, and save it as 182_1x_diff.png. Magnify the difference by 10 times and save the resulting map as 182_10x_diff.png.

In [4]:

def make_adversarial_attack(X, target_y, model):    “””    Generate a fooling image that is close to X, but that the model classifies    as target_y.     Inputs:    – X: Input image; Tensor of shape (1, 3, 224, 224)    – target_y: An integer in the range [0, 1000)    – model: A pretrained CNN     Returns:    – X_fooling: An image that is close to X, but that is classifed as target_y    by the model.    “””        model.eval()        # Initialize our fooling image to the input image    X_fooling = X.clone().detach()    X_fooling.requires_grad = True     # you may change the learning rate and max_iter    learning_rate = 1     ##############################################################################    # TODO: Generate a fooling image X_fooling that the model will classify as   #    # the class target_y. You should perform gradient ascent on the score of the #    # target class, stopping when the model is fooled.                           #    # When computing an update step, first normalize the gradient:               #    #   dX = learning_rate * g / ||g||_2                                         #    ##############################################################################    loss = torch.nn.CrossEntropyLoss()    # The coefficients are tried out by human for several hours    # It is much better by using ones initialization ranther than randn initilization    r = 0.001*torch.ones(1,3,224,224,device=device)    r.requires_grad_(True)    # training the fooling image    for i in range(10000):        X_fooling = X_fooling + r        output = model(X_fooling)        output = torch.nn.functional.softmax(output,dim=1)        # IPNN loss function        cost =  0.00001*torch.norm(r,p=1) + loss(output,target_y)        # BP        cost.backward(retain_graph=True)        # Update the         with torch.no_grad():            r -= learning_rate*r.grad/torch.norm(r.grad,p=2)            # Manually zero the gradients after updating weights            r.grad.zero_()        # Predict y        predict_y = torch.argmax(output)        if predict_y == target_y:            print(‘Interations: %4d    Attack successed！’ %(i),’Predict_y:%d’%(predict_y),’norm(r,2):%f’%(torch.norm(r,p=2).item()))            break    ##############################################################################    #                             END OF YOUR CODE                               #    ##############################################################################    X_fooling = X_fooling.detach()        return X_fooling

In [5]:

############################################################################### TODO: 1. Compute the fooling images for the images under `Project1/images`.##       2. Show the 4 related images of the image ‘182.png’: original image, ##          fooling image, 182_1x_diff.png and2 182_10x_diff.png.              ################################################################################ read imagesimport glob,osfiles = glob.glob(‘images\*.png’)# deal with imagesimages = []for file in files:    print(file)    input_image = PIL.Image.open(file)    input_tensor  = preprocess(input_image).cuda()    target_y = int(file[7:-4])-1-7 # A0206597U    print(target_y)    target_y = torch.tensor([target_y]).cuda()    output_tensor = make_adversarial_attack(input_tensor,target_y,model).cpu()    image = deprocess(output_tensor)    images.append(image)    # save the images    image.save(os.path.join(‘fooling_images’,file[7:-4]+’_’+str(target_y.item())+’.png’))  # plot imagesoriginal_182 = PIL.Image.open(files[1])fig1 = plt.figure()plt.imshow(original_182)plt.title(‘original_182’)fig2 = plt.figure()fooling_182 = images[1]plt.imshow(fooling_182)plt.title(‘fooling_182’)# diff imagestarget_y = int(files[1][7:-4])-1-7 # A0206597Utarget_y = torch.tensor([target_y]).cuda()input_tensor  = preprocess(original_182).cuda()output_tensor = make_adversarial_attack(input_tensor,target_y,model).cpu()difference_1x_182 = deprocess(input_tensor.cpu()-output_tensor,should_rescale=True)difference_10x_182 = deprocess((input_tensor.cpu()-output_tensor)*10,should_rescale=False)fig3 = plt.figure()plt.imshow(difference_1x_182)plt.title(‘difference_1x_182’)fig4 = plt.figure()plt.imshow(difference_10x_182)plt.title(‘difference_10x_182’)# save the diff imagesdifference_1x_182.save(‘182_1x_diff.png’)difference_10x_182.save(‘182_10x_diff.png’)###############################################################################                             END OF YOUR CODE                               ###############################################################################

images\100.png

92

Interations:    7    Attack successed！ Predict_y:92 norm(r,2):2.178698

images\182.png

174

Interations:    5    Attack successed！ Predict_y:174 norm(r,2):2.717995

images\294.png

286

Interations:    4    Attack successed！ Predict_y:286 norm(r,2):2.086930

images\366.png

358

Interations:    3    Attack successed！ Predict_y:358 norm(r,2):2.122715

images\662.png

654

Interations:    5    Attack successed！ Predict_y:654 norm(r,2):2.321405

images\85.png

77

Interations:   12    Attack successed！ Predict_y:77 norm(r,2):2.496330

Interations:    5    Attack successed！ Predict_y:174 norm(r,2):2.717996

Task#2 Class Visualization¶

By starting with a random noise image and performing gradient ascent on a target class, we can generate an image that the network will recognize as the target class. This idea was first presented in [2]; [3] extended this idea by suggesting several regularization techniques that can improve the quality of the generated image.

Concretely, let $I$ be an image and let $y$ be a target class. Let $s_y(I)$ be the score that a convolutional network assigns to the image $I$ for class $y$; note that these are raw unnormalized scores, not class probabilities. We wish to generate an image $I^*$ that achieves a high score for the class $y$ by solving the problem

$$ I^* = \arg\max_I (s_y(I) – R(I)) $$

where $R$ is a (possibly implicit) regularizer (note the sign of $R(I)$ in the argmax: we want to minimize this regularization term). We can solve this optimization problem using gradient ascent, computing gradients with respect to the generated image. We will use (explicit) L2 regularization of the form

$$ R(I) = \lambda \|I\|_2^2 $$

and implicit regularization as suggested by [3] by periodically blurring the generated image. We can solve this problem using gradient ascent on the generated image.

In the cell below, complete the implementation of the create_class_visualization function.

[2] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman, “Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps”, ICLR Workshop 2014

[3] Yosinski et al, “Understanding Neural Networks Through Deep Visualization”, ICML 2015 Deep Learning Workshop

In [6]:

def jitter(X, ox, oy):    “””    Helper function to randomly jitter an image.      Inputs    – X: PyTorch Tensor of shape (N, C, H, W)    – ox, oy: Integers giving number of pixels to jitter along W and H axes        Returns: A new PyTorch Tensor of shape (N, C, H, W)    “””    if ox != 0:        left = X[:, :, :, :-ox]        right = X[:, :, :, -ox:]        X = torch.cat([right, left], dim=3)    if oy != 0:        top = X[:, :, :-oy]        bottom = X[:, :, -oy:]        X = torch.cat([bottom, top], dim=2)    return X

In [7]:

def create_class_visualization(target_y, model, device, **kwargs):    ”’    Generate an image to maximize the score of target_y under a pretrained model.        Inputs:    – target_y: A list of two elements, where the first value is an integer in the range [0, 1000) giving the index of the                class, and the second value is the name of the class.    – model: A pretrained CNN that will be used to generate the image    – dtype: Torch datatype to use for computations        Keyword arguments:    – l2_reg: Strength of L2 regularization on the image    – learning_rate: How big of a step to take    – num_iterations: How many iterations to use    – blur_every: How often to blur the image as an implicit regularizer    – max_jitter: How much to gjitter the image as an implicit regularizer    – show_every: How often to show the intermediate result    ”’    model.to(device)    l2_reg = kwargs.pop(‘l2_reg’, 1e-3)    learning_rate = kwargs.pop(‘learning_rate’, 25)    num_iterations = kwargs.pop(‘num_iterations’, 100)    blur_every = kwargs.pop(‘blur_every’, 10)    max_jitter = kwargs.pop(‘max_jitter’, 16)    show_every = kwargs.pop(‘show_every’, 25)        # Randomly initialize the image as a PyTorch Tensor, and make it requires gradient.    img = torch.randn(1, 3, 224, 224).mul_(1.0).to(device).requires_grad_()     for t in range(num_iterations):        # Randomly jitter the image a bit; this gives slightly nicer results        ox, oy = random.randint(0, max_jitter), random.randint(0, max_jitter)        img.data.copy_(jitter(img.data, ox, oy))         ########################################################################        # TODO: Use the model to compute the gradient of the score for the     #        # class target_y with respect to the pixels of the image, and make a   #        # gradient step on the image using the learning rate. Don’t forget the #        # L2 regularization term!                                              #        # Be very careful about the signs of elements in your code.            #        ########################################################################        scores = model(img)        loss = l2_reg*torch.norm(img,p=2)-2*scores[0,target_y[0]]        loss.backward()        # Update img        with torch.no_grad():            img -= learning_rate*img.grad            # Manually zero the gradients after updating weights            img.grad.zero_()        ########################################################################        #                             END OF YOUR CODE                         #        ########################################################################         # Undo the random jitter        img.data.copy_(jitter(img.data, -ox, -oy))         # As regularizer, clamp and periodically blur the image        for c in range(3):            lo = float(-IMAGENET_MEAN[c] / IMAGENET_STD[c])            hi = float((1.0 – IMAGENET_MEAN[c]) / IMAGENET_STD[c])            img.data[:, c].clamp_(min=lo, max=hi)        if t % blur_every == 0:            blur_image(img.data, sigma=0.5)         # Periodically show the image        if t == 0 or (t + 1) % show_every == 0 or t == num_iterations – 1:            plt.imshow(deprocess(img.data.clone().cpu()))            class_name = target_y[1]            plt.title(‘%s\nIteration %d / %d’ % (class_name, t + 1, num_iterations))            plt.gcf().set_size_inches(4, 4)            plt.axis(‘off’)            plt.show()     return deprocess(img.data.cpu())

Once you have completed the implementation in the cell above, run the following cell to generate an image of a Tarantula:

In [8]:

target_y = [76, “Tarantula”]# target_y = 366 # Gorillaimport pdbout = create_class_visualization(target_y, model, device)

Task#3 Style Transfer¶

Another task which is closely related to image gradients is style transfer which has become a “cool” application in deep learning for computer vision applications. You need to study and implement the style transfer technique presented in the following paper [4] where the general idea is to take two images (a content image and a style image), and produce a new image that reflects the content of one but the artistic “style” of the other.

[4] Gatys, Leon A., Alexander S. Ecker, and Matthias Bethge. “Image style transfer using convolutional neural networks.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.

Below is an example.

Compute the loss¶

To perform style transfer, you will need to first formulate a special loss function that matches the content and style of each respective image in the feature space, and then perform gradient descent on the pixels of the image itself.

The loss function contains two parts: content loss and style loss. Read the paper [4] for details about the losses and implement them below.

In [9]:

def content_loss(content_weight, content_current, content_original):    “””    Compute the content loss for style transfer.        Inputs:    – content_weight: Scalar giving the weighting for the content loss.    – content_current: features of the current image; this is a PyTorch Tensor of shape      (1, C_l, H_l, W_l).    – content_target: features of the content image, Tensor with shape (1, C_l, H_l, W_l).        Returns:    – scalar content loss    “””        ##############################################################################    # TODO: Implement content loss function                                      #    # Note: It should not be very much code (less than 10 lines)                 #    ##############################################################################    loss = content_weight*torch.sum((content_current-content_original)**2)    return loss    ##############################################################################    #                             END OF YOUR CODE                               #    ##############################################################################  def gram_matrix(features):    “””    Compute the normalized Gram matrix from features.    The Gram matrix will be used to compute style loss.        Inputs:    – features: PyTorch Tensor of shape (N, C, H, W) giving features for      a batch of N images.        Returns:    – gram: PyTorch Tensor of shape (N, C, C) giving the      normalized Gram matrices for the N input images.    “””        ##############################################################################    # TODO: Implement the normalized Gram matrix compuation function             #    # Note: It should not be very much code (less than 10 lines)                 #    ##############################################################################    N,C,H,W = features.size()    features = torch.reshape(features,(N,C,-1))    gram = torch.bmm(features,features.permute(0,2,1))/float(C*H*W)    return gram    ##############################################################################    #                             END OF YOUR CODE                               #    ##############################################################################  def style_loss(feats, style_layers, style_targets, style_weights):    “””    Computes the style loss at a set of layers.        Inputs:    – feats: list of the features at every layer of the current image.    – style_layers: List of layer indices into feats giving the layers to include in the      style loss.    – style_targets: List of the same length as style_layers, where style_targets[i] is      a PyTorch Variable giving the Gram matrix of the source style image computed at      layer style_layers[i].    – style_weights: List of the same length as style_layers, where style_weights[i]      is a scalar giving the weight for the style loss at layer style_layers[i].          Returns:    – style_loss: A PyTorch Tensor holding a scalar giving the style loss.    “””        ##############################################################################    # TODO: Implement style loss function                                        #    # Note: It should not be very much code (less than 10 lines)                 #    ##############################################################################    loss = 0    for i,layer in enumerate(style_layers):        cur_gram = gram_matrix(feats[layer])        loss += style_weights[i]*torch.sum((cur_gram-style_targets[i])**2)/4    return loss    ##############################################################################    #                             END OF YOUR CODE                               #    ##############################################################################

Putting them together¶

With these loss functions, you can now build your style transfer model. Implement the function below to perform style transfer. To test the model, you can use the content and style images that we have provided in Project1/style, or improvise using any image you like. Please save your output images in the Project1/style folder.

Design and carry out some experiments (on your own!) to analyse how the choice of layers and the weights will influence the output image. Write down your observations and analysis in the Markdown cell provided below.

In [146]:

def style_transfer(content_image, style_image, content_layer, content_weight,                   style_layers, style_weights, max_iter):    “””    Run style transfer!    You may first resize the image to a small size for fast computation.        Inputs:    – content_image: filename of content image    – style_image: filename of style image    – content_layer: an index indicating which layer to use for content loss    – content_weight: weighting on content loss    – style_layers: list of indices indicating which layers to use for style loss    – style_weights: list of weights to use for each layer in style_layers    – max_iter: max iterations of gradient updates        Returns:    – output_image: an image with content from the content_image and     style from the style image    “””    ##############################################################################    # TODO: Implement the function for style transfer.                           #    ##############################################################################    def extract_features(data,model):        feats = []        prev_feat = data        for module in model.features:                next_feat = module(prev_feat)                feats.append(next_feat)                prev_feat = next_feat        return feats    # remove white noise    def variance_loss(image):        loss = 0        # row        loss += torch.sum( ( image[:,:,1:,:] – image[:,:,:-1,:] )**2 )         # column        loss += torch.sum( ( image[:,:,:,1:] – image[:,:,:,:-1] )**2 )        return loss    # Content image    content_img = preprocess(PIL.Image.open(content_image)).to(device)    feats = extract_features(content_img,model)    content_target = feats[content_layer].clone()    # Style image    style_img = preprocess(PIL.Image.open(style_image)).to(device)    feats = extract_features(style_img,model)    style_target = []    for layer in style_layers:        style_target.append(gram_matrix(feats[layer].clone()))    # Transfer image    tran_img = content_img.clone().requires_grad_()    # Training    optimizer = torch.optim.Adam([tran_img],lr=1.0)    # Plot input images    f, axarr = plt.subplots(1,2)    axarr[0].axis(‘off’)    axarr[1].axis(‘off’)    axarr[0].set_title(‘Content Source Img.’)    axarr[1].set_title(‘Style Source Img.’)    axarr[0].imshow(deprocess(content_img.cpu()))    axarr[1].imshow(deprocess(style_img.cpu()))    plt.show()    plt.figure()    for i in range(max_iter):        # Enhancing contrast ratio        tran_img.data.clamp_(-1.5, 1.5)        # Init grad        optimizer.zero_grad()        feats = extract_features(tran_img, model)        # Compute loss        c_loss = content_loss(content_weight, feats[content_layer], content_target)        s_loss = style_loss(feats, style_layers, style_target, style_weights)        v_loss = variance_loss(tran_img)        total_loss = 0.2*c_loss + s_loss + 0.1*v_loss        # BP        total_loss.backward()        optimizer.step()    print(‘Iteration {}’.format(i))    plt.axis(‘off’)    plt.imshow(deprocess(tran_img.data.cpu()))    deprocess(tran_img.data.cpu()).save(‘style/engineering-the_scream.png’)    plt.show()    ##############################################################################    #                             END OF YOUR CODE                               #    ##############################################################################

In [147]:

############################################################################### TODO: 1. Choose one pair of images under ‘Project1/style’, and finish the  # #          neural style transfer task by calling the style_transfer function.##       2. Show the 3 related images: content image, style image and the     # #          generated style-transferred image.                                ################################################################################ Content layer comaparing params1 = {    ‘content_image’ : ‘style/engineering.jpg’,    ‘style_image’ : ‘style/the_scream.jpg’,    ‘content_layer’ : 3,    ‘content_weight’ : 0.1,     ‘style_layers’ : (1, 3, 4, 6, 7),    ‘style_weights’ : (20000, 10000,500, 12, 1),    ‘max_iter’ : 100}style_transfer(**params1)###############################################################################                             END OF YOUR CODE                               ###############################################################################

Iteration 99

As the content layer index increases, the transmitted image will look more like “style_image”. In addition, the ratio of $ \ alpha / \ beta $ also controls the similarity with “content_image” or “style_image”. The larger the loss caused by the style layer, the more it looks like a style image.