Summaries Artificial Intelligence Revolutionizing Food Recognition with Pie Torch and Google Storage | Nutrify #1

This is a summary of a YouTube video "Copying Tesla's data engine, but with food images | nutrify #1" by Daniel Bourke!

4.0 (64 votes)

Automating the training process for a food recognition app using Pie Torch Image Models, Weights and Biases, and Google Storage to improve accuracy and reduce CPU, battery, and RAM usage.

• 🤖
  00:00

  Automating the training process for a food recognition app using Pie Torch Image Models, Weights and Biases, and Google Storage.
• 📊
  03:05

  Track data input and output to create a trained model artifact stored on Weights and Biases.
• 🤖
  05:21

  Download model, use dataset, write code, export/show results of model confusion to predict on single image and save details to dictionary.
• 🤔
  07:31

  Using improved prediction code, we are plotting the most wrong samples to compare predicted classes to true classes and improve labels for a better model.
• 🔧
  12:02

  I have created a visual interface to re-label images and am slowly putting together the pieces of the data flywheel.
• 🤖
  16:10

  Using a model to label data, then updating labels and training a smaller model to deploy, or using a similarity model to figure out similar classes, can help with confusing examples.
• 📊
  19:31

  Updated dataset labels, trained model, and logged results; got a boost in training accuracy and lower training loss; refining scripts and tracking results on Weights and Biases next week.
• 💻
  23:57

  Rebuilt the Nutrified demo app to reduce CPU, battery, and RAM usage by 10x, but currently it is only detecting lemons, so Josh will investigate next week.

Detailed summary

• 🤖
  00:00

  Automating the training process for a food recognition app using Pie Torch Image Models, Weights and Biases, and Google Storage.

  • This week I'm making the training script reproducible, using the Pie Torch Image Models library on GitHub, tracking experiments and results with Weights and Biases, and using the latest data from Google Storage.
  • Nutrify is creating an album on a user's phone to store photos taken with the app and use the data to provide more detailed food information.
  • Run python train.pi to call the training script with a bunch of hacked together code.
• 📊
  03:05

  Track data input and output to create a trained model artifact stored on Weights and Biases.

  • We can click into weights and biases to see how the model is performing after 10 epochs, and then iterate over the data set to create a data flywheel.
  • Given a certain dataset, tracking what data goes into a model, labels, and what comes out, allows for the creation of a trained model artifact stored on Weights and Biases.
• 🤖
  05:21

  Download model, use dataset, write code, export/show results of model confusion to predict on single image and save details to dictionary.

  • Download best model from weights and biases, use same dataset, write code for model to make predictions, export/show results of model confusion.
  • This function predicts on a single image, saves the details of the prediction to a dictionary, and returns a list of dictionaries with the image path, prediction label, probability, and pred class.
• 🤔
  07:31

  Using improved prediction code, we are plotting the most wrong samples to compare predicted classes to true classes and improve labels for a better model.

  • Updated prediction code revealed model was randomly guessing, so now plotting most wrong samples based on improved prediction code.
  • Comparing the predicted classes to the true classes of the most wrong predictions with the highest prediction probability can help improve labels and get a better model.
  • Create a table from the most wrong data frame, turn it into a dictionary, and get a list of dictionaries of the top 100 most wrong samples.
  • We turn posix paths into strings, use weights and biases to turn images into viewable objects, and log a table with columns to the run to view as an interactive data frame.
• 🔧
  12:02

  I have created a visual interface to re-label images and am slowly putting together the pieces of the data flywheel.

  • We can improve the accuracy of our model by loading images into labeling software and updating the true labels.
  • I figured out how to convert my labels from a CSV into the required Json format for Label Studio.
  • A hacky code is used to create a series of HTML strings which can be used to set up a labeling interface with instructions.
  • I have a visual interface to re-label images and slowly putting together the pieces of the data flywheel.
• 🤖
  16:10

  Using a model to label data, then updating labels and training a smaller model to deploy, or using a similarity model to figure out similar classes, can help with confusing examples.

  • Using a large model to label data, then updating labels and training a smaller model to deploy to devices, or using a similarity model to figure out similar classes, can help with confusing examples like nectarine and peach.
  • The model can predict blueberries but also returns information for similar fruits like strawberries, raspberries, and blackberries.
  • Merged updated labels from Label Studio with original annotations in Google Cloud Storage bucket.
• 📊
  19:31

  Updated dataset labels, trained model, and logged results; got a boost in training accuracy and lower training loss; refining scripts and tracking results on Weights and Biases next week.

  • I updated 60 labels in the dataset, then trained the model and logged the results.
  • Re-add the reference to the updated annotations path on Google Cloud Storage to let weights and biases track the new version.
  • We didn't get better results, but we got a boost in training accuracy and a slightly lower training loss.
  • Next week will be refining scripts to train and predict, and tracking results on Weights and Biases.
• 💻
  23:57

  Rebuilt the Nutrified demo app to reduce CPU, battery, and RAM usage by 10x, but currently it is only detecting lemons, so Josh will investigate next week.

  • Nutrified demo was built as an AR app using the AR camera, but it was capping out at 30 frames per second.
  • Rebuilt the app using Swift UI and UI Kit to reduce CPU usage, battery usage, and RAM usage by 10x.
  • The vision request is detecting food or not food using two models, but currently it is only outputting lemons, so next week Josh will have to go back and investigate the photos being sent to the models.