Running the model programmatically with Python and integrating Ollama with LangChain — building on your Ollama setup.
Lesson 3.1 (continued): Running Ollama Models Programmatically & Integrating with LangChain
Step 6: Running Ollama Models via Python
Ollama can be interacted with programmatically to integrate its models into your own applications.
Using Python to run prompts on Ollama models
-
Install Python HTTP client (optional):
Ollama supports a simple HTTP API. You can use Python’s
requestslibrary to send prompts.pip install requests -
Example Python script to send prompts to Ollama:
import requests def run_ollama_model(prompt, model_name='mistral'): url = f"http://localhost:11434/api/run/{model_name}" payload = {"prompt": prompt} response = requests.post(url, json=payload) if response.status_code == 200: return response.json().get("text", "") else: return f"Error: {response.status_code} - {response.text}" if __name__ == "__main__": prompt_text = "Explain the importance of local AI models." output = run_ollama_model(prompt_text) print("Model output:", output) -
Note:
Ensure Ollama’s HTTP API server is running locally (usually on port11434). Check Ollama documentation for starting the API server if it’s not running by default.
Step 7: Integrating Ollama with LangChain
LangChain is a framework to build AI applications that can use different LLMs as backends. You can use Ollama models as a custom LLM in LangChain.
Setting up LangChain to use Ollama
-
Install LangChain:
pip install langchain -
Create a custom Ollama wrapper class for LangChain:
Since LangChain may not have an official Ollama wrapper yet, you can build a simple custom LLM class that calls Ollama via Python.
from langchain.llms.base import LLM from typing import Optional, List, Mapping import requests class OllamaLLM(LLM): def __init__(self, model_name: str = "mistral", api_url: str = "http://localhost:11434/api/run/"): self.model_name = model_name self.api_url = f"{api_url}{model_name}" def _call(self, prompt: str, stop: Optional[List[str]] = None) -> str: payload = {"prompt": prompt} response = requests.post(self.api_url, json=payload) response.raise_for_status() return response.json().get("text", "") @property def _identifying_params(self) -> Mapping[str, str]: return {"model_name": self.model_name} @property def _llm_type(self) -> str: return "ollama" -
Use this custom OllamaLLM with LangChain:
from langchain.chains import ConversationChain from langchain.prompts import PromptTemplate from langchain.memory import ConversationBufferMemory # Initialize Ollama LLM wrapper ollama_llm = OllamaLLM(model_name="mistral") # Create a simple conversation prompt template template = """The following is a friendly conversation. Human: {input} AI:""" prompt = PromptTemplate(input_variables=["input"], template=template) # Setup conversation memory memory = ConversationBufferMemory() # Create conversation chain conversation = ConversationChain(llm=ollama_llm, prompt=prompt, memory=memory) # Interact with the agent user_input = "What is LangChain?" response = conversation.run(user_input) print("AI response:", response)
Summary of this extension:
-
You can interact with Ollama models locally through a Python HTTP API.
-
By wrapping the Ollama API calls inside a custom LangChain LLM class, you can use Ollama models seamlessly within LangChain agents.
-
This approach enables you to build powerful AI agents that run fully locally with privacy and zero API costs.
47
