DygnosTech

Demo Video: here

Keep your side effects aside.

Inspiration💡

Today with the number of medications people have to take and considering the multiple doctors one consults for different problems, they inevitably face adverse side effects from these medicines. DygnosTech is a solution for both doctors and patients to make their life easier and not fall prey to these side effects and allow doctors to make better decisions while prescribing a medicine.

We thus ended up building DygnosTech through which doctors and patients can leverage the power of being able to predict the likely side effects a drug would have on the patient. This can greatly help doctors make better decisions as well as for the patient to understand what side effects the drugs he takes have.

What it does🔎

DygnosTech is a web app that holds promise for both doctors and patients.

• From the doctor’s perspective: DygnosTech takes in the names of the drug the doctor is about to prescribe and runs them through the Machine Learning model we created to identify the most likely side effects for the patient given their past medical history. Having This information alongside the doctor while prescribing a medicine could be very helpful!

• From the patient perspective: DygnosTech intakes an image of prescriptions from the patient, which are then parsed with some OCR techniques and instantly displays a graph mapping the medicines with their possible side effects. The analysis of side effects the patient is most likely to have from the medicine is done through a Machine Learning model we created and is also personalized to the patient. This can greatly help a patient keep track of the side effects. The user also gets the option of sending himself a summary of the most likely side effects.

How we built it🔨

We majorly divide this project into two sub-projects:

DygnosTech Machine Learning Model

Throughout the course of this project we made use of 2 different Machine Learning models for different use cases:

1. Identifying Side Effects from the disease

This model was one of the main aspects and novelty of our project. We wanted to be able to identify the side effects a drug would cause on a particular user. DygnosTech Machine Learning Model after thorough experimentation and also taking into account the data we have, we build on top of the Graph Attention Network model using it as a backbone for our model. We first through a simple lookup identify the molecular structure of the drug and then trained our model on the graph data (i.e. molecules) we got from doing so.

Having ended up with quite a large model, which we spent time training on and understanding the quirks for, optimizing this model was of the essence. We were able to introduce some optimizations to the model. Finally, we also decided to perform the inference for the model using WebAssembly (Wasm) and serialized the WebAssembly module which paired with some other optimization we make greatly improved the model performance.

1. Identifying drugs from a prescription

We also used Machine Learning, especially character recognition techniques to identify the text written in the prescription and then parsed it to find the drugs being prescribed. These outputs after being parsed would go in the aforementioned model and then be shown to the patient.

DygnosTech Web App

The DygnosTech Web App was built using Streamlit which simplified for us the labyrinth of managing data-heavy applications like ours, state management for a large model, and more. The web app is written in Streamlit allowed us to introduce optimizations easily for the data and memory-intensive tasks we want to be able to run for this application. We used graphviz to plot some crucial tree graphs. HTML and CSS helped us make our web pages more visually attractive and interactive.

Challenges we ran into⚠️

• Identifying and finding the right kind of data for the machine-learning model.
• Our first pass at integrating the model into the web application was quite bulky as well as took some time to infer from. Optimizing the model and finding or modifying the optimization techniques that work for our use cases was one of the big challenges. Integrating Wasm in this environment proved to be a quite difficult task however well worth the performance gains.
• Implementing HTML and CSS code in Streamlit was new for us. We used one of the features of Streamlit, components.html to complete this task in an elegant and efficient manner.
• Merging Twilio in our project was a very critical and fun step in our progress.

Accomplishments that we’re proud of🥇

• After multiple attempts and thousands of training images, we were able to train and deploy a well-working graph neural network model which accurately identifies side effects a person is likely to have
• We are quite happy that we were able to integrate the models with the UI and do so efficiently
• We were also able to teach each other in this hackathon, some of us had no experience with Machine Learning while some were not able to make great UIs

What we learned🧠

• Most of our project members were not experienced with techniques in the area of Machine Learning or modern machine learning models like the one we used.
• All our project members are first-year students at the University of Toronto, but our combined experience helped us complete this project in the allotted time. Our belief in teamwork also increased during the course of this hackathon. Most of all, it solidified our view that copious amounts of clean data are what drive the best artificial intelligence systems.

What’s next for Weed Detech💭

• At the moment for being able to execute our model as a WebAssembly module we employed some serialization techniques to make it work in our environment. Though web assembly is quite secure, I think we should work more on the security aspects.
• Right now our application only tells you about the side effects and we also had some ideas on how we could add information or suggestions on overcoming these side effects.

Check it out here: GitHub Repo