Skip to main contentHomepage

Talk with Open Source LLM

Interactive chat wrapper for testing and comparing open-source LLMs like LLaMA-3, Mistral, and Gemma

Visit Website / SpaceView Source / Demo
Talk with Open Source LLM

Overview

A clean, interactive chat wrapper built with Streamlit and LangChain that provides a simple playground for conversing with and comparing open-source foundation models including Meta LLaMA-3, Mistral 7B, and Google Gemma.

The Problem

Exploring and evaluating open-weight Large Language Models often involves complex setup hurdles:

  • Heavy Local Hardware Requirements: Running multi-gigabyte open-weight models locally requires dedicated high-VRAM GPU hardware that isn't always readily accessible.
  • Overcomplicated Boilerplate: Setting up inference pipelines, tokenizers, and conversation memory just to test basic conversational prompts creates unnecessary friction.
  • Model Comparison Friction: Quickly switching between different open model architectures to evaluate response quality and prompt compliance requires a unified testing interface.

The Solution

  • Streamlined Model Wrapper: Built an intuitive Streamlit interface powered by LangChain to interact with open-source models hosted on Hugging Face inference endpoints.
  • On-the-Fly Model Switching: Allows users to easily switch between LLaMA-3, Mistral, and Gemma to evaluate reasoning and response variations.
  • Conversation State & Streaming: Implemented real-time token streaming and conversational buffer memory for natural, multi-turn interactions.
  • Hugging Face Spaces Deployment: Packaged and deployed as a live cloud application for instant, zero-setup access.

Key Takeaways

  • Simple & Accessible: Offers an easy way to experiment with state-of-the-art open models without local hardware barriers.
  • Rapid Prototyping: Serves as a versatile foundation for testing prompt engineering techniques across different model architectures.