Deploying Palmyra-E Model Step-by-Step
Deploying Palmyra-E (30B) Model Step-by-Step
Prerequisites:
- A virtual machine (VM) with 4A100 80GB GPU to run the model
- Python 3.6 or higher
- Git
Follow these steps to deploy the Palmyra-E model:
Step 1: Create a virtual environment
1.1. Open a terminal and navigate to your desired working directory.
1.2. Run the following commands to create a virtual environment and activate it:
python -m venv palmyra-e-env
source palmyra-e-env/bin/activate # On Linux and macOS
palmyra-e-env\Scripts\activate # On Windows
Step 2: Clone the Palmyra-E repository
2.1. Run the following command to clone the Palmyra-E GitHub repository:
git clone https://github.com/writerai/palmyra-e.git
2.2. Change your working directory to the cloned repository folder:
cd palmyra-e
Step 3: Install required packages
3.1. Run the following command to install the necessary packages:
pip install -r requirements.txt
Step 4: Run the Palmyra-E model
4.1. Create a Python script (e.g., run_palmyra_e.py
) and paste the provided usage code into the script.
4.2. Save the script and run it using the following command:
python run_palmyra_e.py
4.3. The script will download the Palmyra-E model and generate a response based on the given instruction. You should see the output on the terminal.
Step 5: (Optional) Deploy the model as a REST API
5.1. Install FastAPI and Uvicorn by running the following command:
pip install fastapi uvicorn
5.2. Create a new Python script (e.g., palmyra_e_api.py
) and paste the following code into the script:
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
class InstructionInput(BaseModel):
instruction: str
@app.post("/generate/")
async def generate_response(input_data: InstructionInput):
instruction = input_data.instruction
# Replace the following line with the Palmyra-E model code
response = f"Generated response for: {instruction}"
return {"response": response}
5.3. Replace the response = f"Generated response for: {instruction}"
line with the Palmyra-E model code.
5.4. Save the script and run the API server using the following command:
uvicorn palmyra_e_api:app --host 0.0.0.0 --port 8000
5.5. The REST API is now running and accessible at http://localhost:8000
. You can send POST requests to the /generate/
endpoint with an instruction to receive generated responses.
With these steps, you have successfully deployed the Palmyra-E model on your virtual machine. You can further customize the deployment to suit your specific requirements.
Updated 28 days ago