Testing Mulitple Local LLM with LM Studio

For LM Studio, this automated workflow tests multiple local LLMs by integrating with Sticky Note, LangChain, and Google Sheets. It captures response metrics, analyzes readability, and tracks performance over time, allowing users to evaluate model outputs effectively. With 21 nodes, it streamlines the process of comparing LLMs, ensuring concise and clear responses while providing optional data logging for comprehensive analysis.

7/4/2025
21 nodes
Complex
rktiztdblvr6uzsgw3xdiseiujd7xgbamanualcomplexsticky notelangchaingooglesheetssplitoutadvancedapiintegration
Categories:
Data Processing & AnalysisManual TriggeredComplex Workflow
Integrations:
Sticky NoteLangChainGoogleSheetsSplitOut

Target Audience

This workflow is ideal for:
- Data Analysts: Individuals who analyze and report on model performance metrics.
- Researchers: Those conducting studies on language model outputs and their readability.
- Developers: Engineers looking to integrate local LLMs into applications for testing purposes.
- Educators: Teachers or trainers wanting to evaluate the readability of content generated by language models for educational materials.
- Product Managers: Managers overseeing AI projects that require performance tracking and reporting.

Problem Solved

This workflow addresses the need for automated testing and analysis of language model outputs. It provides a systematic approach to:
- Retrieve and manage multiple LLMs.
- Analyze the readability and effectiveness of responses from these models.
- Track performance metrics over time, allowing for informed decision-making and model adjustments.

Workflow Steps

  • Trigger: The workflow starts manually when a chat message is received.
    2. Model Retrieval: It queries the local LLM server to fetch the list of available models.
    3. System Prompt Addition: A guiding prompt is added to ensure responses are concise and appropriate for a 5th-grade reading level.
    4. Response Analysis: The chat input is sent to the selected models, and their responses are analyzed for various metrics, including:
    - Readability Score
    - Word Count
    - Sentence Count
    - Average Sentence Length
    - Average Word Length
    5. Time Tracking: The workflow captures the start and end times of the model execution to calculate total time spent.
    6. Data Preparation: The analyzed metrics are structured for reporting.
    7. Google Sheets Integration: Results are saved to a Google Sheet for easy access and further analysis.

    • Customization Guide

    Users can customize this workflow by:
    - Modifying the Base URL: Update the Base URL in the workflow to match your local LLM server's IP.
    - Adjusting Model Settings: Change parameters like temperature, top P, and presence penalty in the Run Model with Dynamic Inputs node to fit specific testing criteria.
    - Editing Prompts: Tailor the System Prompt to focus on different aspects of the model's output based on the desired outcomes.
    - Altering Google Sheet Structure: Customize the Google Sheets node to include additional metrics or modify existing column headers as needed.
    - Adding More Nodes: Extend the workflow by integrating additional functionality, such as notifications or further data processing steps.