Deploying Palmyra-E Model Step-by-Step

Suggest Edits

Deploying Palmyra-E (30B) Model Step-by-Step

Prerequisites:

  • A virtual machine (VM) with 4A100 80GB GPU to run the model
  • Python 3.6 or higher
  • Git

Follow these steps to deploy the Palmyra-E model:

Step 1: Create a virtual environment
1.1. Open a terminal and navigate to your desired working directory.
1.2. Run the following commands to create a virtual environment and activate it:

python -m venv palmyra-e-env
source palmyra-e-env/bin/activate  # On Linux and macOS
palmyra-e-env\Scripts\activate     # On Windows

Step 2: Clone the Palmyra-E repository
2.1. Run the following command to clone the Palmyra-E GitHub repository:

git clone https://github.com/writerai/palmyra-e.git

2.2. Change your working directory to the cloned repository folder:

cd palmyra-e

Step 3: Install required packages
3.1. Run the following command to install the necessary packages:

pip install -r requirements.txt

Step 4: Run the Palmyra-E model
4.1. Create a Python script (e.g., run_palmyra_e.py) and paste the provided usage code into the script.

4.2. Save the script and run it using the following command:

python run_palmyra_e.py

4.3. The script will download the Palmyra-E model and generate a response based on the given instruction. You should see the output on the terminal.

Step 5: (Optional) Deploy the model as a REST API
5.1. Install FastAPI and Uvicorn by running the following command:

pip install fastapi uvicorn

5.2. Create a new Python script (e.g., palmyra_e_api.py) and paste the following code into the script:

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class InstructionInput(BaseModel):
    instruction: str

@app.post("/generate/")
async def generate_response(input_data: InstructionInput):
    instruction = input_data.instruction
    # Replace the following line with the Palmyra-E model code
    response = f"Generated response for: {instruction}"
    return {"response": response}

5.3. Replace the response = f"Generated response for: {instruction}" line with the Palmyra-E model code.

5.4. Save the script and run the API server using the following command:

uvicorn palmyra_e_api:app --host 0.0.0.0 --port 8000

5.5. The REST API is now running and accessible at http://localhost:8000. You can send POST requests to the /generate/ endpoint with an instruction to receive generated responses.

With these steps, you have successfully deployed the Palmyra-E model on your virtual machine. You can further customize the deployment to suit your specific requirements.

Updated 28 days ago