Title of Presentation:
“Multi-modal Large Language Models for Prompt Guided Image Analysis”
Description:
Large language models coupled with a vision module, such as in GPT4V allow for text prompts augmented with images. The goal of our research is to get an answer to a question about an image using a minimum number of labeled images. In particular, we want to classify whether a mouse brain’s slices from the hippocampus imaged at 20X resolution have been treated with saline or trimethylitn (TMT). The staining was Iba1-immunostain for microglia cells. The goal is to do as well as a deep learned model using prompts of text and a minimal number of slices. Using well crafted prompts with a few images from 11 cases (1270 images), the same accuracy was achieved as with a custom deep learned model based on VGG16. Further, the predictions came with explanations. This talk will discuss how the prompts were constructed, explanations extracted and how they were corrected in an active prompting cycle. The approach is general and can (and is) being applied to other datasets.
About the Speaker:
Lawrence O. Hall is a Distinguished University Professor in the Department of Computer Science and Engineering at the University of South Florida and the co-Director of the Institute for Artificial Intelligence + X. He received a Ph.D. in Computer Science from Florida State University (1986) and a B.S. in Applied Mathematics from Florida Institute of Technology (1980). He is a fellow of the IEEE, AAAS, AIMBE, IAPR, and AAIA. He received the 2021 Fuzzy Pioneer award from the IEEE CI Society. He received the Norbert Wiener award in 2012 and the Joseph Wohl award in 2017 from the IEEE SMC Society. He was the IEEE VP Publications 2021-2. He is a past President of the IEEE Systems, Man and Cybernetics Society, former EIC of what is now the IEEE Transactions on Cybernetics. He is on the editorial boards of the Proceedings of the IEEE and IEEE Spectrum.
His research interests lie in learning from big data, distributed machine learning, medical image understanding, bioinformatics, pattern recognition, modeling imprecision in decision making, and integrating AI into image processing. He continues to explore un and semi-supervised learning using scalable fuzzy approaches. He has authored or co-authored over 100 publications in journals, as well as many conference papers and book chapters.