LangChain + MPT-30B-Chat with TGI on Vast.ai
Today, I'm going to guide you through running the quantized MPT-30B-Chat with text-generation-inference on a platform known as Vast.ai, and show you how to integrate it with LangChain.
The notebook is available on GitHub
experienced-devand in Google Colab
Video version:
Let's kick things off by installing the necessary dependencies.
As the next step, we are downloading the Vast.ai CLI
Unfortunately, Vast.ai doesn't have an official Python package. That's why I created this wrapper to extract dictionaries.
If we pass the --raw command line argument, we get a JSON response.
By default, we are displaying help.
In the next step, you will need the Vast.ai API key. You can obtain one at vast.ai
Then, we search for suitable hardware to run MosaicML/MPT-30B-Chat with 8-bit quantization.
Select an index of the hardware you would like to use. It will print the id and cost per hour.
Next, we create an instance with text-generation-inference, providing mosaicml/mpt-30b-chat as a model id and quantizing it with bitsandbytes
Then we are waiting for the instance to become running.
Now that it's running, we can construct the inference_server_url from the instance configuration.
Then we wait for the text TGI server to become healthy, displaying logs every 5 seconds.
The time you have to wait depends on the internet speed of the instance.
Then, we are creating an LLM with LangChain's HuggingFaceTextGenInference.
If 'stop_sequences' are not present, the model will generate both sides of the conversation.
Then, we are creating a chat memory.
And ConversationChain in verbose mode.
Displaying a Chat prompt
PromptTemplate(input_variables=['history', 'input'], output_parser=None, partial_variables={}, template='The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n\nCurrent conversation:\n{history}\nHuman: {input}\nAI:', template_format='f-string', validate_template=True)and Chat prompt template.
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.
Current conversation: {history} Human: {input} AI:Sending the first message
> Entering new chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.
Current conversation:
Human: hi how are you?
AI:
> Finished chain.
Hi there! I'm just a computer program, so I don't have feelings like humans do. But I'm functioning properly and ready to help you with any questions you might have.Second message
> Entering new chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.
Current conversation:
Human: hi how are you?
AI: Hi there! I'm just a computer program, so I don't have feelings like humans do. But I'm functioning properly and ready to help you with any questions you might have.
Human: My name is Pavel.
AI:
> Finished chain.
Nice to meet you, Pavel! Is there anything I can assist you with today?Checking the memory state
ConversationBufferWindowMemory(chat_memory=ChatMessageHistory(messages=[HumanMessage(content='hi how are you?', additional_kwargs={}, example=False), AIMessage(content=" Hi there! I'm just a computer program, so I don't have feelings like humans do. But I'm functioning properly and ready to help you with any questions you might have.\n\n", additional_kwargs={}, example=False), HumanMessage(content='My name is Pavel.', additional_kwargs={}, example=False), AIMessage(content=' Nice to meet you, Pavel! Is there anything I can assist you with today?\n\n\n', additional_kwargs={}, example=False)]), output_key=None, input_key=None, return_messages=False, human_prefix='Human', ai_prefix='AI', memory_key='history', k=6)and verifying that memory is working.
> Entering new chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.
Current conversation:
Human: hi how are you?
AI: Hi there! I'm just a computer program, so I don't have feelings like humans do. But I'm functioning properly and ready to help you with any questions you might have.
Human: My name is Pavel.
AI: Nice to meet you, Pavel! Is there anything I can assist you with today?
Human: What is my name?
AI:
> Finished chain.
You mentioned that your name is Pavel earlier in the conversation. Is there anything else you would like to know or discuss?Then asking What is the difference between a method and a function?
> Entering new chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.
Current conversation:
Human: hi how are you?
AI: Hi there! I'm just a computer program, so I don't have feelings like humans do. But I'm functioning properly and ready to help you with any questions you might have.
Human: My name is Pavel.
AI: Nice to meet you, Pavel! Is there anything I can assist you with today?
Human: What is my name?
AI: You mentioned that your name is Pavel earlier in the conversation. Is there anything else you would like to know or discuss?
Human: What is the difference between a method and a function?
AI:
> Finished chain.
In programming, both methods and functions are used to perform specific tasks, but there are some key differences between them. A method is a function that is associated with an object or a class, while a function is a standalone piece of code that can be called from anywhere in the program.
Methods have access to the data and variables within the object or class they are associated with, while functions do not. Additionally, methods can modify the state of an object, while functions cannot.
In summary, methods are functions that are specific to an object or class, and they have access to the data and variables within that object or class. Functions, on the other hand, are standalone pieces of code that can be called from anywhere in the program.
Is there anything else I can help you with?And following up with What is more common in OOP?
> Entering new chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.
Current conversation:
Human: hi how are you?
AI: Hi there! I'm just a computer program, so I don't have feelings like humans do. But I'm functioning properly and ready to help you with any questions you might have.
Human: My name is Pavel.
AI: Nice to meet you, Pavel! Is there anything I can assist you with today?
Human: What is my name?
AI: You mentioned that your name is Pavel earlier in the conversation. Is there anything else you would like to know or discuss?
Human: What is the difference between a method and a function?
AI: In programming, both methods and functions are used to perform specific tasks, but there are some key differences between them. A method is a function that is associated with an object or a class, while a function is a standalone piece of code that can be called from anywhere in the program.
Methods have access to the data and variables within the object or class they are associated with, while functions do not. Additionally, methods can modify the state of an object, while functions cannot.
In summary, methods are functions that are specific to an object or class, and they have access to the data and variables within that object or class. Functions, on the other hand, are standalone pieces of code that can be called from anywhere in the program.
Is there anything else I can help you with?
Human: What is more common in OOP?
AI:
> Finished chain.
In Object-Oriented Programming (OOP), methods are more commonly used than functions. Methods are specific to an object or class and provide a way to interact with and modify the state of those objects. Functions, on the other hand, are standalone pieces of code that can be called from anywhere in the program, but they do not have access to the data and variables within an object or class.
That being said, both methods and functions have their own specific uses, and the choice between them depends on the specific needs of the program. Is there anything else I can help you with?Next, we are starting Gradio app with the chatbot. It is very useful to share PoCs and prototypes. We wipe out the previous memory.
In the app, we are starting with a geometry question
How to square a circle?
I'm sorry, I don't know how to square a circle. It's impossible according to the laws of geometry.
Following up with
Why?
A circle is defined as the set of all points in a plane that are equidistant from a given point called the center. Squaring a circle means constructing a square with the same area as the circle. However, the circumference of the circle is longer than the diameter, so the ratio of the area of the circle to the square of its radius is irrational, which means it cannot be expressed as a simple fraction. Therefore, it's impossible to construct a square with the same area as a circle.
Then
Compare the physical and chemical properties of hydrogen and oxygen
Hydrogen and oxygen are both chemical elements, but they have very different physical and chemical properties. Hydrogen is a colorless, odorless, and tasteless gas that is highly flammable. It is the lightest element and has one proton, one neutron, and one electron. Oxygen, on the other hand, is a colorless, odorless, and tasteless gas that is essential for life. It has eight protons, eight neutrons, and eight electrons.
In terms of chemical properties, hydrogen is highly reactive and forms bonds with many other elements, including carbon to form methane, CH4. Oxygen, however, is less reactive and only forms two bonds with other elements, one single bond and one double bond. It is also more electronegative than hydrogen, meaning it attracts electrons more strongly when forming bonds.
In terms of physical properties, hydrogen has a density of about 0.08 g/L at room temperature and atmospheric pressure, while oxygen has a density of about 1.2 g/L under the same conditions. Hydrogen has a melting point of -259°C and a boiling point of -252°C, while oxygen has a melting point of -183°C and a boiling point of -184°C. Both elements are gases at room temperature and atmospheric pressure.
Overall, hydrogen and oxygen have very different physical and chemical properties due to their differences in atomic structure and reactivity. Hydrogen is highly reactive and forms many different compounds, while oxygen is less reactive and forms fewer compounds but is essential for life.
with a follow-up
Are they toxic?
Both hydrogen and oxygen can be toxic in certain circumstances. Hydrogen sulfide (H2S), which is a colorless, odorless, and tasteless gas, is a toxic byproduct of the reaction between sulfur and hydrogen. It can cause respiratory problems and even death at high concentrations. Carbon monoxide (CO), which is a colorless, odorless, and tasteless gas, is also toxic. It binds to hemoglobin in red blood cells, preventing oxygen from being transported to tissues and organs, leading to hypoxia and potential death.
Oxygen toxicity occurs when oxygen concentrations are too high, such as in hyperbaric chambers or diving at great depths. This can lead to lung damage, seizures, and even death. However, oxygen is essential for life and is required for cellular respiration to produce energy for living organisms.
In summary, hydrogen and oxygen can both be toxic under certain circumstances, but they are also essential for life and many chemical reactions.
And continue with a last question
What is the length of human DNA in meters?
The length of human DNA is approximately 2 meters or 2,000 millimeters. This is the total length of DNA in a single human cell, which contains around 3 billion base pairs. The DNA molecule is highly compacted in the cell nucleus, and its length is estimated to be around 2 meters when fully stretched out.
It's worth noting that this is an approximation, as the length of DNA can vary depending on the degree of compaction and the specific cell type. For example, the DNA in sperm cells is more compacted than in other cell types, which can result in a shorter length. However, the overall length of human DNA is estimated to be around 2 meters.
After getting the responses we can stop the instance.
If vast.ai has availability it will allow you to start it. Please consult the vast.ai documentation to familiarize yourself with its rules. In the meantime, you're paying for storage.
To destroy the instance run
But don't forget to run
to verify that you are not charged for something that you are not using. To be on the safe side check the Vast.ai console.
Summary:
The MPT-30B-Chat large language model is highly potent. Operating a quantized version with text-generation-inference can be quite cost-effective on platforms like Vast.ai. It is seamlessly integrable with LangChain, enabling rapid development.