HackDavis 2024 #

Our Github

Mission: Reduce car related injuries and deaths

Our project for https://hackdavis.io/event 2024.

Use Intel Developer Cloud to train AI 🚀
Performance gains (reduce PyTorch train time by 4 minutes) with bfloat16 and ipex ⚡
Tensorflow performance gains (train time going from 2:23s to 1:14s) Almost twice as fast with ipex!
Use ipex with PyTorch in IDC
Make three different OpenCV models for lane detection, human obstacle detection, and driver alertness detection

Parts of the whole system #

Lane Detection #

Use a forward facing camera and OpenCV to recognize lanes and alert the driver if they do not stay within their lane.

Driver Alertness Detection #

Use a driver facing camera and OpenCV to detect if the driver is awake and paying attention to the road

Blind Spot Detection #

Use PyTorch and Intel Developer Cloud Notebook to detect pedestrians walking infront and next to the car and alert the driver if they get too close. We were able to use Intel’s ipex, PyTorch plugin, and bfloat16 to reduce the training time by 4 entire minutes. We used ipex from Intel AI and the Intel PyTorch plugin to leverage Intel AMX.

ipex example usage #

import torch
import intel_extension_for_pytorch as ipex
from engine import train_one_epoch, evaluate
import datetime

# ...

params = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.SGD(
    params,
    lr=0.005,
    momentum=0.9,
    weight_decay=0.0005
)

lr_scheduler = torch.optim.lr_scheduler.StepLR(
    optimizer,
    step_size=3,
    gamma=0.1
)

model = get_model_instance_segmentation(num_classes)

model, optimizer = ipex.optimize(model, optimizer=optimizer, dtype=torch.float32)
model = model.to(device)

Optimization Experiment:

Start time: 2024-04-27 16:27:36.983459
End time: 2024-04-27 16:47:38.238154

datetime.timedelta(seconds=1201, microseconds=254695)

Start time: 2024-04-27 17:46:16.191538
End time: 2024-04-27 18:02:06.233684

datetime.timedelta(seconds=950, microseconds=42146)

Start time: 2024-04-27 17:02:22.302090
End time: 2024-04-27 17:18:17.402227

datetime.timedelta(seconds=955, microseconds=100137)

As you can see, intel ipex and bfloat16 saved us 4 entire minutes for this model. Imagine a much larger model and the type of time and compute cost savings that could be achived.

Intel LeaderBoard #

We submitted a custom bfloat16 model to the intel leaderboard. Here is the model fine tuned from Gemma

Use of LLMs and implications of hallucinations #

We used LLMs to generate and extend data that we used to fine tune our text directions Gemma model. Hallucinations for directions in this case could cause people to go to places that do not exist. This could be fixed by using existing databases of locations (Like a mapping API) to insure that the LLM is always directing someone to a real place.