PA3: Digit Recognition with Keras with Dense and CNNs
Learning Objectives
After completing this activity, students should be able to:
- Build a simple Neural Network in Keras
- Implement a classifier that codes output in one-hot encoding
- Evaluate different network architectures
- Analyze and plot computational cost versus accuracy
Partners
This assignment may be completed individually or in pairs.
If you are doing this in pairs, you must notify me at the beginning
of the project.
My expectation for pairs is that both members are
actively involved, and take full responsibility for all aspects
of the project. In other words, I expect that you are either
sitting or virtually together to work, and not that you are splitting
up tasks to be completed separately. If both members of the
group are not able to fully explain the code to me, then
this does not meet this expectation.
Implementation
You will need to build 4 Keras models for this PA:
- build two NN that utilize dense connections. Name these files
that implement these networks m1_dense.py, and m2_dense.py
- build two NN that utilizes a CNN
architecture. Name these files m1_cnn.py and
m2_cnn.py
Required Analysis for Each Model
For each of your model, you
must justify your choice of hyperparameters,
including:
- number of epochs
- Network designs (nodes, dropout layers, etc)
Justification involves generating supportive evidence from running your model (and not speculation).
For example, plots of the training/validation loss and/or accuracy comparing the impact of adding nodes, layers,
and/or dropout layers. Above all, for each model,
you must make an argument (with supporting evidence) that you are
not overfitting.
Compare Model Performance on the Test Data
Run your 4 models on the test data and create:
- a single plot that compares their accuracy
- a table that shows for each model the:
- number of parameters
- accuracy
For your best model, save your model to a h5 and upload it
to Autolab (along with the preprocessing file). Your model will be on some
top secret test data and the results will be posted as a scoreboard.
Support Files
The following files are available:
| Filename |
Description |
| keras_model_template.py |
Template file for each of your Keras models |
| keras_pre.py |
A template for your preprocessing class. This function is
responsible for recoding and reshaping your data as required
for your model. You will need a preprocessing file for
each of the 4 models you create (named m1_pre.py, m2_pre.py, ...). |
| keras_test_digit_model.py |
Unit test program (same as used by Autolab).
Your programs MUST work when called from
this program. |
| MNIST_X_train.npy |
Xtrain
Training data, containing 60k gray scaled
images (shape is 60000, 28, 28).
images. |
| MNIST_y_train.npy |
ytrain
Labels for the 60k images in the training set. |
| MNIST_X_test.npy |
Xtest
Test data. 10k images (10000, 28, 28). |
| MNIST_X_test.npy |
ytest
Labels for the 10k images in the test set. |
Submission
This PA has the following required submissions:
| Deliverable |
Location to Submit |
Description |
| mlnnauto.zip |
Autolab |
A zip file containing: with your best model and preprocessing code. which should be named:
- Your best saved model (name this file BestModel.h5. Do NOT include the
.py file in this zip.
- The cooresponding preprocessing file (name this file BestPre.py.
|
| cs445_pa3.pdf |
Canvas |
Discussion of your model's performance (accuracy) that you evaluated and supporting evidence
of your hyperparameters. Your model's performance should be plotted and illustrated
per epoch. |
| mlnncode.zip |
Canvas |
A zip containing the following files:
- m1_dense.py and m1_pre.py (the preprocessing file for m1_dense.py)
- m2_dense.py and m2_pre.py (the preprocessing file for m2_dense.py)
- m3_cnn.py and m3_pre.py (the preprocessing file for m3_cnn.py)
- m4_cnn.py and m4_pre.py (the preprocessing file for m4_cnn.py)
Tested and graded using KerasTestDigitModel and supporting plots for
hyper-parameter justification (plots contained in cs445_pa3.pdf). |
Grading
Grades will be calculated according to the following
distribution.
| Criterion |
Percentage |
Description |
| Overall Readability/Style |
10% |
Your code should follow PEP8 conventions. It should be
well documented and well organized. |
| Dense Models |
40% |
You dense models must run (they will be verified using
keras_test_digit_model.py and the preprocessing file).
The PDF must contain a section that describes the architecture of each model of the dense models
and then compares their
performance using plots (for training, validation, and test data).
The training and validation results must be shown per epoch.
Compare/contrast the two models performance
and support why you believe that one of the models is not overfitting (1 to 2 paragraphs). |
| CNN Models |
40% |
Same as the dense models (but for the two CNN models). |
| Summary of all 4 Models |
5% |
Summarize/compare all 4 models and identify which one
performs the best (summary section with 1 paragraph and
supporting plot comparing performance) in the PDF. The plots
required are listed in the compare model performance on
the test data section.
|
| Model Benchmark on Autolab |
5% |
This tests your model's performance on the hidden data. Having your
code run and submitted to Autolab will result in full credit for this section. |