A Rust and Python neural network implemented without using ML/AI frameworks.

Background

I’ve recently wanted to get back into AI and since I love the Rust programming language, I thought of learning more about AI while deepening my knowledge with Rust.

I’ve built a few AI models in the past and took a few AI courses (at the university and online), but had forgotten a lot about the specifics of neural networks.

Implementation

I’ve started by following this tutorial:

https://towardsdatascience.com/math-neural-network-from-scratch-in-python-d6da9f29ce65

I started by implementing the neural network in Rust, then ran into problems that I wasn’t able to solve initially. I then decided to implement it in Python, which I was able to do.

The Python implementation very closely resembles the code from the article, but I’ve tried to make it more modular and easier to understand for the Rust implementation.

Performance

Both have comparable performance due to them being very similar. The Rust implementation is slightly faster for the XOR example, but the MNIST example is slightly faster in Python. I’m guessing that numpy is more optimized than my Rust implementation and spreads the workload across multiple threads.

In addition, both implementations are simple and naïve - they do not contain batching or other methods to speed up the training process.

XOR

The XOR example is a very simple neural network that learns the XOR function. It has 2 inputs, 2 layers, and 1 output. It is trained with 4 examples. The training is very fast and the accuracy is close to 100%.

The implementations have similar result/accuracy. I took the time to train and execute on my M1 Max Mac where the Rust implementation is way faster than the Python implementation:

MNIST

The MNIST example is a more complex neural network that learns to recognize handwritten digits. It has 784 inputs (28 x 28), 3 layers, and 10 outputs. It is trained with 1,000 examples, since the training is extremely slow - it could be trained with 60,000 examples. The accuracy is around 90%.

The implementations have again similar result/accuracy. I took the time to train and execute on my M1 Max Mac where the Python implementation is way faster than the Rust implementation:

All in all, it’s been a fun experience and I’ve learned a lot about neural networks and Rust. I’m looking forward to learning more about AI and Rust. Hoping to touch more newer topics like GANs, Transformers, LLMs, RAGs, etc.