Anyscale

Anyscale is a fully-managed Ray platform, on which you can build, deploy, and manage scalable AI and Python applications

This example goes over how to use LangChain to interact with Anyscale Endpoint.

import os

os.environ["ANYSCALE_API_BASE"] = ANYSCALE_API_BASE
os.environ["ANYSCALE_API_KEY"] = ANYSCALE_API_KEY

from langchain.llms import Anyscale
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain

template = """Question: {question}

Answer: Let's think step by step."""

prompt = PromptTemplate(template=template, input_variables=["question"])

llm = Anyscale(model_name=ANYSCALE_MODEL_NAME)

llm_chain = LLMChain(prompt=prompt, llm=llm)

question = "When was George Washington president?"

llm_chain.run(question)

With Ray, we can distribute the queries without asynchronized implementation. This not only applies to Anyscale LLM model, but to any other Langchain LLM models which do not have _acall or _agenerate implemented

prompt_list = [
    "When was George Washington president?",
    "Explain to me the difference between nuclear fission and fusion.",
    "Give me a list of 5 science fiction books I should read next.",
    "Explain the difference between Spark and Ray.",
    "Suggest some fun holiday ideas.",
    "Tell a joke.",
    "What is 2+2?",
    "Explain what is machine learning like I am five years old.",
    "Explain what is artifical intelligence.",
]

import ray


@ray.remote(num_cpus=0.1)
def send_query(llm, prompt):
    resp = llm(prompt)
    return resp


futures = [send_query.remote(llm, prompt) for prompt in prompt_list]
results = ray.get(futures)