Mac Machine Learning Speed Test

Blog post | Video walkthrough

A collection of simple scripts focused on benchmarking the speed of various machine learning models on Apple Silicon Macs (M1, M2, M3).

Scripts should also ideally work with CUDA (for benchmarking on other machines/Google Colab).

Note: Scripts are not designed to achieved state-of-the-art results (e.g. accuracy), they are designed to be as simple as possible to run out of the box. Most are examples straight from PyTorch/TensorFlow docs I've tweaked for specific focus on MPS (Metal Performance Shaders - Apple's GPU acceleration framework) devices + simple logging of timing. They are scrappy and likely not the best way to do things, but they are simple and easy to run.

Experiment Overview

The focus of these experiments is to get a quick benchmark across various ML problems and see how the Apple Silicon Macs perform.

The focus is on hardware comparison rather than framework to framework comparison and measuring speed rather than accuracy.

This repo contains code/results for the following experiments:

1. PyTorch Computer Vision (CIFAR100 image classification)
2. PyTorch Computer Vision (Food101 image classification)
3. PyTorch Natural Langua2ge Processing (NLP text classification)
4. TensorFlow Computer Vision (CIFAR100 image classication)
5. TensorFlow Computer Vision (Food101 image classification)
6. TensorFlow Natural Language Processing (NLP text classification)
7. LlamaCPP LLM test (text generation)
8. Geekbench ML (inference-only benchmarks)

While the focus is on Apple Silicon Macs, I've included my own deep learning PC (NVIDIA TITAN RTX) as well as a Google Colab free tier instance for comparison.

Getting Setup

If you have a brand new machine, you'll need to setup a few things before running the experiments.

The following steps will get you ready to go for all experiments (and many future machine learning experiments).

However, if you've already got conda, feel free to skip to the next section.

Base environment setup

1. Install homebrew (or run xcode-select --install in terminal and skip to next step)

Go to https://brew.sh/ and follow the main instructions on the front page.

Run the commands on the homebrew webpage in the terminal and follow the instructions when they appear.

2. Install miniforge to get conda: https://github.com/conda-forge/miniforge

brew install miniforge

or

Download Miniforge3 for macOS ARM64 from: https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh

3. Run the following commands in terminal with Miniforge3 downloaded into the ~/Downloads folder:

chmod +x ~/Downloads/Miniforge3-MacOSX-arm64.sh
sh ~/Downloads/Miniforge3-MacOSX-arm64.sh

4. Follow the steps, for example, answer "yes", "yes", "ok" etc and then initialize conda to see if it works.

source ~/miniforge3/bin/activate

5. Important: Restart terminal and check conda is working.

If conda is working, you should have a (base) at the start of your terminal prompt.

For example: (base) daniel@Daniels-MacBook-Pro-3 ~ %

Setting up for machine learning tests

1. Clone this repo.

git clone https://github.com/mrdbourke/mac-ml-speed-test.git 

2. Change into the repo directory.

cd mac-ml-speed-test

3. Create conda environment.

conda create --prefix ./env python=3.10

Note: You could also use conda create --name some-env-name python=3.10 but I prefer --prefix as it's more explicit.

4. Check conda environments.

conda env list

5. Activate newly created conda environment.

conda activate ./env

6. Install necessities/helper packages.

Note: This may have a few extra packages that aren't 100% needed for speed tests but help to have (e.g. JupyterLab, PrettyTable).

conda install -c conda-forge pip pandas numpy matplotlib scikit-learn jupyterlab langchain prettytable py-cpuinfo tqdm

Install and Test PyTorch/Hugging Face Transformers

• Apple guide to installing PyTorch.
• PyTorch guide to installing PyTorch.
• Hugging Face Guides to Install Transformers, Datasets, Evaluate, Accelerate.

conda install pytorch::pytorch torchvision -c pytorch

Note: MPS (Metal Performance Shaders, aka using the GPU on Apple Silicon) comes standard with PyTorch on macOS, you don't need to install anything extra. MPS can be accessed via torch.mps, see more notes in the PyTorch documentation.

Test PyTorch Computer Vision (CIFAR100)

Experiment details:

Model	Dataset	Image Size	Epochs	Num Samples	Num Classes	Problem Type
ResNet50	CIFAR100	32x32x3	5	50,000 train, 10,000 test	100	Image Classification

Example usage of pytorch_test_computer_vision_cifar100.py for 1 epoch and batch size of 32:

python pytorch_test_computer_vision_cifar100.py --epochs=1 --batch_sizes="32"

Batch sizes can be a comma-separated list of batch sizes, e.g. "32, 64, 128, 256".

Default behaviour is to test for 5 epochs and batch sizes of "16, 32, 64, 128, 256, 512, 1024".

The following:

python pytorch_test_computer_vision_cifar100.py

Is equivalent to:

python pytorch_test_computer_vision_cifar100.py --epochs=5 --batch_sizes="16, 32, 64, 128, 256, 512, 1024"

Results will be saved to results/results_pytorch_cv/[file_name].csv where file_name is a combination of information from the experiment (see pytorch_test_computer_vision_cifar100.py for details).

Test PyTorch Computer Vision (Food101)

Experiment details:

Model	Dataset	Image Size	Epochs	Num Samples	Num Classes	Problem Type
ResNet50	Food101	224x224x3	5	75,750 train, 25,250 test	101	Image Classification

Note: Download Hugging Face Datasets to download Food101 dataset.

python -m pip install datasets

Example usage of pytorch_test_computer_vision_food101.py for 1 epoch and batch size of 32:

python pytorch_test_computer_vision_food101.py --epochs=1 --batch_sizes="32"

Batch sizes can be a comma-separated list of batch sizes, e.g. "32, 64, 128, 256".

Default behaviour is to test for 3 epochs and batch sizes of "32, 64, 128".

The following:

python pytorch_test_computer_vision_food101.py

Is equivalent to:

python pytorch_test_computer_vision_food101.py --epochs=3 --batch_sizes="32, 64, 128"

Results will be saved to results/results_pytorch_cv/[file_name].csv where file_name is a combination of information from the experiment (see pytorch_test_computer_vision_food101.py for details).

Test PyTorch Natural Language Processing (NLP)

Experiment details:

Model	Dataset	Sequence Size	Epochs	Num Samples	Num Classes	Problem Type
DistilBERT (fine-tune top 2 layers + top Transformer block)	IMDB	512	5	25,000 train, 25,000 test	2	Text Classification

Note: The pytorch_test_nlp.py uses Hugging Face Transformers/Datasets/Evaluate/Accelerate to help with testing. If you get into ML, you'll likely come across these libraries, they are very useful for NLP and ML in general. The model loaded from Transformers uses PyTorch as a backend.

python -m pip install transformers datasets evaluate accelerate

Example usage of pytorch_test_nlp.py for 1 epoch and batch size of 32:

python pytorch_test_nlp.py --epochs=1 --batch_sizes="32"

Batch sizes can be a comma-separated list of batch sizes, e.g. "32, 64, 128, 256".

Default behaviour is to test for 3 epochs and batch sizes of "16, 32, 64, 128, 256, 512" (note: without 24GB+ of RAM, running batch sizes of 256+ will likely error, for example my M1 Pro with 18GB of VRAM can only run "16, 32, 64, 128" and fails on 256 with the model/data setup in python_test_nlp.py).

The following:

python pytorch_test_nlp.py

Is equivalent to:

python pytorch_test_nlp.py --epochs=3 --batch_sizes="16, 32, 64, 128, 256, 512"

Results will be saved to results/results_pytorch_nlp/[file_name].csv where file_name is a combination of information from the experiment (see pytorch_test_nlp.py for details).

Install and Test TensorFlow

For more on running TensorFlow on macOS, see Apple's developer guide.

Note: Install TensorFlow Datasets to access Food101 dataset with TensorFlow.

python -m pip install tensorflow
python -m pip install tensorflow-metal  
python -m pip install tensorflow_datasets

Note: TensorFlow can be run on macOS without using the GPU via pip install tensorflow, however, if you're using an Apple Silicon Mac, you'll want to use the Metal plugin for GPU acceleration (pip install tensorflow-metal).

After installing tensorflow-metal and running the scripts, you should see something like:

2023-12-06 12:22:02.016745: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:117] Plugin optimizer for device_type GPU is enabled.

Test TensorFlow Computer Vision (CIFAR100)

Experiment details:

Model	Dataset	Image Size	Epochs	Num Samples	Num Classes	Problem Type
ResNet50	CIFAR100	32x32x3	5	50,000 train, 10,000 test	100	Image Classification

Example usage of tensorflow_test_computer_vision_cifar100.py for 1 epoch and batch size of 32:

python tensorflow_test_computer_vision_cifar100.py --epochs=1 --batch_sizes="32"

Batch sizes can be a comma-separated list of batch sizes, e.g. "32, 64, 128, 256".

Default behaviour is to test for 5 epochs and batch sizes of "16, 32, 64, 128, 256, 512, 1024".

The following:

python tensorflow_test_computer_vision_cifar100.py

Is equivalent to:

python tensorflow_test_computer_vision_cifar100.py --epochs=5 --batch_sizes="16, 32, 64, 128, 256, 512, 1024"

Results will be saved to results/results_tensorflow_cv/[file_name].csv where file_name is a combination of information from the experiment (see tensorflow_test_computer_vision_cifar100.py for details).

Test TensorFlow Computer Vision (Food101)

Experiment details:

Model	Dataset	Image Size	Epochs	Num Samples	Num Classes	Problem Type
ResNet50	Food101	224x224x3	5	75,750 train, 25,250 test	101	Image Classification

Example usage of tensorflow_test_computer_vision_food101.py for 1 epoch and batch size of 32:

python tensorflow_test_computer_vision_food101.py --epochs=1 --batch_sizes="32"

Batch sizes can be a comma-separated list of batch sizes, e.g. "32, 64, 128".

Default behaviour is to test for 3 epochs and batch sizes of "32, 64, 128".

The following:

python tensorflow_test_computer_vision_food101.py

Is equivalent to:

python tensorflow_test_computer_vision_food101.py --epochs=3 --batch_sizes="32, 64, 128"

Results will be saved to results/results_tensorflow_cv/[file_name].csv where file_name is a combination of information from the experiment (see tensorflow_test_computer_vision_food101.py for details).

Test TensorFlow Natural Language Processing (NLP)

Experiment details:

Model	Dataset	Sequence Size	Epochs	Num Samples	Num Classes	Problem Type
SmallTransformer (custom)	IMDB	200	5	25,000 train, 25,000 test	2	Text Classification

Example usage of tensorflow_test_nlp.py for 1 epoch and batch size of 32:

python tensorflow_test_nlp.py --epochs=1 --batch_sizes="32"

Batch sizes can be a comma-separated list of batch sizes, e.g. "32, 64, 128, 256".

Default behaviour is to test for 3 epochs and batch sizes of "16, 32, 64, 128".

The following:

python tensorflow_test_nlp.py

Is equivalent to:

python tensorflow_test_nlp.py --epochs=3 --batch_sizes="16, 32, 64, 128"

Results will be saved to results/results_tensorflow_nlp/[file_name].csv where file_name is a combination of information from the experiment (see tensorflow_test_nlp.py for details).

Install and Test LlamaCPP (Llama 2 LLM test)

Experiment details:

Model	Task	Num Questions	Num Answers	Total Generations
Llama 2 7B .gguf format	Text Generation	20	5	20*5 = 100

• See: https://llama-cpp-python.readthedocs.io/en/latest/install/macos/ (note: this focuses on macOS install, I haven't tested with CUDA)

CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 python -m pip install llama-cpp-python

After installing llama-cpp-python, you will need a .gguf format model from Hugging Face.

• Download a model from Hugging Face with .gguf extension, e.g. https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/blob/main/llama-2-7b-chat.Q4_0.gguf →