Lesson 3.1 (continued): Running Ollama Models Programmatically & Integrating with LangChain

Running the model programmatically with Python and integrating Ollama with LangChain — building on your Ollama setup.


Lesson 3.1 (continued): Running Ollama Models Programmatically & Integrating with LangChain


Step 6: Running Ollama Models via Python

Ollama can be interacted with programmatically to integrate its models into your own applications.

Using Python to run prompts on Ollama models

  1. Install Python HTTP client (optional):

    Ollama supports a simple HTTP API. You can use Python’s requests library to send prompts.

    pip install requests
    
  2. Example Python script to send prompts to Ollama:

    import requests
    
    def run_ollama_model(prompt, model_name='mistral'):
        url = f"http://localhost:11434/api/run/{model_name}"
        payload = {"prompt": prompt}
        response = requests.post(url, json=payload)
        if response.status_code == 200:
            return response.json().get("text", "")
        else:
            return f"Error: {response.status_code} - {response.text}"
    
    if __name__ == "__main__":
        prompt_text = "Explain the importance of local AI models."
        output = run_ollama_model(prompt_text)
        print("Model output:", output)
    
  3. Note:
    Ensure Ollama’s HTTP API server is running locally (usually on port 11434). Check Ollama documentation for starting the API server if it’s not running by default.


Step 7: Integrating Ollama with LangChain

LangChain is a framework to build AI applications that can use different LLMs as backends. You can use Ollama models as a custom LLM in LangChain.

Setting up LangChain to use Ollama

  1. Install LangChain:

    pip install langchain
    
  2. Create a custom Ollama wrapper class for LangChain:

    Since LangChain may not have an official Ollama wrapper yet, you can build a simple custom LLM class that calls Ollama via Python.

    from langchain.llms.base import LLM
    from typing import Optional, List, Mapping
    import requests
    
    class OllamaLLM(LLM):
        def __init__(self, model_name: str = "mistral", api_url: str = "http://localhost:11434/api/run/"):
            self.model_name = model_name
            self.api_url = f"{api_url}{model_name}"
    
        def _call(self, prompt: str, stop: Optional[List[str]] = None) -> str:
            payload = {"prompt": prompt}
            response = requests.post(self.api_url, json=payload)
            response.raise_for_status()
            return response.json().get("text", "")
    
        @property
        def _identifying_params(self) -> Mapping[str, str]:
            return {"model_name": self.model_name}
    
        @property
        def _llm_type(self) -> str:
            return "ollama"
    
  3. Use this custom OllamaLLM with LangChain:

    from langchain.chains import ConversationChain
    from langchain.prompts import PromptTemplate
    from langchain.memory import ConversationBufferMemory
    
    # Initialize Ollama LLM wrapper
    ollama_llm = OllamaLLM(model_name="mistral")
    
    # Create a simple conversation prompt template
    template = """The following is a friendly conversation.
    
    Human: {input}
    AI:"""
    prompt = PromptTemplate(input_variables=["input"], template=template)
    
    # Setup conversation memory
    memory = ConversationBufferMemory()
    
    # Create conversation chain
    conversation = ConversationChain(llm=ollama_llm, prompt=prompt, memory=memory)
    
    # Interact with the agent
    user_input = "What is LangChain?"
    response = conversation.run(user_input)
    print("AI response:", response)
    

Summary of this extension:

  • You can interact with Ollama models locally through a Python HTTP API.

  • By wrapping the Ollama API calls inside a custom LangChain LLM class, you can use Ollama models seamlessly within LangChain agents.

  • This approach enables you to build powerful AI agents that run fully locally with privacy and zero API costs.


 

47