Introduction to Deep Learning, ESPCI


This course is an introduction to deep-learning approach with lab sessions in pytorch (python module). The goal is to understand the data-driven approach and to be able to efficiently experiment with deep-learning on real data.

News

  • Course update : "confinement" !
  • Are you sure you registered your team for the project (see the drive) ?

Modified organisation (in French)

Suite aux conséquences du Coronavirus, l'organisation du cours "Introduction au Deep-Learning" est modifiée. Nous allons nous concentrer sur les projets à rendre. La date est fixée désormais au lundi 18/05 au plus tard. Bien sûr vous pouvez le rendre avant la 4 mai (période de début de stage) mais étant données les circonstances, il peut être profitable d'avoir quelques jours en plus pour terminer la rédaction ou quelques expériences.

Sur la feuille partagée (vous devez le connaître mais aller voir quand même). il y a la liste des groupes et des sujets. Si vous n'êtes pas inscrit.e.s sur cette feuille, c'est désormais urgent de le faire.

Visio-conférence sur le projet le 23/03 matin

Lundi 23/03 matin, plusieurs visio-conférences vont être organisées pour discuter des projets. Pour cela, nous allons utiliser zoom. C'est facile à installer et gratuit: https://us04web.zoom.us/. Il y aura une réunion zoom par sujet de projet.

L'objectif est d'avoir une réunion zoom avec toutes les personnes traitant du même projet pour partager les informations, difficultés et solutions. Cela veut dire de votre côté de vous organiser pour être connecté.e.s à ce moment là et être à l'heure. Je vais envoyer un planning bientôt.

Chaque réunion commencera par un tour de parole où chaque groupe résumera brièvement où il en est, et les questions qu'il souhaite aborder. Pour préparer ces réunions, il est important de préparer une liste de questions et de me l'envoyer en avance d'ici vendredi (inclus).

D'ici là, n'hésitez pas à me contacter par mail, et on peut faire une visio si besoin.

A rendre pour le 18/05

A rendre, un rapport au format pdf et une archive avec le code. Le rapport d'une dizaine de pages, doit décrire:

  • les données et la tâche abordées, en particulier les hypothèses, pré-traitement et quelles sont les entrées et sorties
  • les problématiques abordées et les solutions mise en pratique
  • les résultats décrits, commentés et analysés.

L'archive contenant le code ne doit contenir que le code et pas les données. Si vous le jugez nécessaire, il est possible d'échanger les données par drive interposé. Une solution est de m'envoyer l'archive avec votre rapport par mail, ou de m'envoyer l'adresse d'un dépôt.

The resources / drive

Look at this drive for the slides and the material of lab sessions.

Expected schedules

It starts in january 2020 (the 6th). The course are scheduled on monday, starting at 8:30 in the morning.

The first part:

  • 6/01, course: introduction and basics on machine learning
    • The main tasks
    • Objective function
    • Optimisation with gradient descent
  • 13/01, lab session : machine learning and python, first steps
    • numpy and matplotlib
    • logistic regression by hand
  • 20/01, course on feed-forward neural networks
    • Multi-class classification
    • From linear to non-linear classification
    • The feed-forward architecture and the back-propagation algorithm
  • 27/01, course on deep-learning
    • Deep networks
    • Drop out, regularization and other tricks
    • pytorch overview
  • 03/02, lab session : deep-learning in pytorch
  • 10/02, course: Embedding and sequence processing with convolution (part 1)

The second part:

  • 02/03, course: Convolutional (part 2) and recurrent deep-networks
    • reminder on 1D convolution
    • 2D convolution for image processing
    • Recurrent net
  • 09/03, lab session: Sequence classification with 1D convolution
  • 23/03, course: TBA
  • 30/03, course: TBA

Projects

Here, you can find a list of possible projects. Feel free to interact with me. For some of them, just ask me the data, otherwise a link is provided. Of course, you can also propose a project.

Project list

  • Reconstruction of the vorticity field of a flow behind a cylinder from a handful sensors on the cylinder surface
Velocity fields Sea surface temperature

See below for more details

  • Chemistry: Predict the standard density of pure fluids, using a newly compiled database. From SMILES description, how can we predict density ? Ask me for the data and tools.
  • Classify sleep and arousal stages from physiological signals including: electroencephalography (EEG), electrooculography (EOG), electromyography (EMG), electrocardiology (EKG), and oxygen saturation (SaO2). See the challenge page for more details
  • Classify, from a single short ECG lead recording (between 30 s and 60 s in length), whether the recording shows normal sinus rhythm, atrial fibrillation (AF), an alternative rhythm, or is too noisy to be classified: The challenge page.
  • Using seismic signals to predict the timing of laboratory earthquakes.
  • Quantum-mechanical molecular energies prediction from the raw molecular geometry: see the QM7 database.
  • Classify Molecule polarization: the data comes from time-lapse fluorescence microscopy images of the bacterium Pseudomonas fluorescens SBW25. Each image is an individual bacterial cell. These bacteria produce a molecule called pyoverdin which is naturally fluorescent, so the images show the distribution of this molecule inside the cells. We have discovered that there are two distribution patterns of this molecule: homogeneous, or accumulated at the cell pole ("polarized").

Ocean data

The datasets from this site provide different kind of measures. It is possible to consider different project. The datasets are in special format. To read it in python, you can use two python librairies:

  • netCDF4 is used to read the data
  • basemap to plot the earth map

If you use anaconda, you can install them with conda install. A notebook (quickly written) is available on the course drive (in datasets/Ocean) to show how to read the data.

You can also read them with scipy as proposed below but I don't understand the resulting data encoding. So I would not recommend it.

from scipy.io import netcdf
nc = netcdf.netcdf_file("./Downloads/sst.wkmean.1981-1989.nc",'r')
# Acces to the variable list 
print(nc.variables)
Out:   OrderedDict([('lat', <scipy.io.netcdf.netcdf_variable at 0x10422c208>),
               ('lon', <scipy.io.netcdf.netcdf_variable at 0x10422c1d0>),
               ('time', <scipy.io.netcdf.netcdf_variable at 0x10422c4e0>),
               ('time_bnds', <scipy.io.netcdf.netcdf_variable at 0x10422c550>),
               ('sst', <scipy.io.netcdf.netcdf_variable at 0x10422c6d8>)])
# Get the time array 
time = nc.variables['time']
# This is a "netcdf" variable, to access the data as numpy array
print(time.data)
# You can also try with 'sst' to get the temperature

Several application can be considered as a project:

  • Like in this paper, you can learn to reconstruct the map given some known points (for example randomly selected). In the paper they only use a "shallow" decoder, of course you can try it, but the project should explore a more advanced architecture like convolutional network (or in this case: deconvolution).
  • For a given number of measures, can we optimize the place of this measures to improve the reconstruction ?
  • And what if the sensors move ?
  • A time prediction can be also an interesting task. This means predicting the sst given the week before.

Pytorch

How to save a model to share and resume training

The key structure in pytorch is the state dict for state dictionnary. The learnable parameters (i.e. weights and biases) of an torch.nn.Module model are contained in the model’s parameters (accessed with model.parameters()). A state dict is simply a Python dictionary object that maps each layer to its parameter tensor. The basic idea is to save and load this dictionnary to easily deal with parameters in a consistent way. Note that only layers with learnable parameters (convolutional layers, linear layers, etc.).

To resume a training process you need to restore the model in its full state, but also the optimizer. Optimizer objects (torch.optim) also have a state dict, which contains information about the optimizer’s state, as well as the hyperparameters used.

Creating state dict a model and an optimizer

Assume you have a model like

# Define model
class TheModelClass(nn.Module):
    def __init__(self):
        super(TheModelClass, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

# Initialize model
model = TheModelClass()

model_state_d = model.state_dict()
# iter on the keys of the dictionnary
for param_tensor in model_state_d:
    print(param_tensor, "\t", model_state_d[param_tensor].size())

# You want to access to a specific tensor of parameter
print(model_state_d["conv1.bias"]) 

You can do the same for the optimizer.

# Initialize optimizer
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

# Print optimizer's state_dict
print("Optimizer's state_dict:")
for var_name in optimizer.state_dict():
    print(var_name, "\t", optimizer.state_dict()[var_name])

In general you don't need to look at the details yourself. You may want to save these parameters in a single file to restore your training session.

Saving a training session

First, we can create an object to gather and store all the necessary data (parameters). And then we can write it in a file. Later we will be able to reload it.

# Here we create a dictionnary to store what we need: 
# 1/ the epoch number
# 2/ the paramters of the model
# 3/ the paramters of the optimizer
# 4/ the loss function we used
tosave_data = 
{
            'epoch': epoch_counter,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'loss': loss,
}
# Write a pickle file using pytorch 
torch.save(tosave_data, "/the/path/to/my/file")

And that's it !

Resume training

Now we can read a file to restore the training session.

# First create objects of the good types 
prev_model = TheModelClass()
prev_optimizer = optim.SGD(model.parameters(),lr=0.1)

# Load the pickle file to read the dictionnary from the file
checkpoint = torch.load("/the/path/to/my/file")

# restore the model and optimizer states: 
prev_model.load_state_dict(checkpoint['model_state_dict'])
prev_optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']