HuggingFace Resources

Loading resources into your Nosana Deployment is essential to making sure the deployment is ready to go as fast as possible.

To facilitate this you can use the resources property of the Nosana Job Definition.
At the moment there are two primary built in providers that you can use to load in an external resource.

1. HuggingFace
2. S3

HuggingFace is the gold standard for hosting Machine Learning Models, providing a vast repository of open-source models. S3, on the other hand, is the gold standard for storing and retrieving any kind of data, offering robust and scalable storage solutions.

With the combination of these two storage services, you can create Nosana Deployments efficiently, and with tools you know and love.

HuggingFace

Most open source models that you know and love are hosted and tracked on HuggingFace.
That's why downloading HuggingFace models has first class support in the Nosana Ecosystem.

As an example, let's say you want to use the following LLM: TinyLlama/TinyLlama-1.1B-Chat-v1.0.

In order to load this model into the Docker Runtime of your Nosana deployment you will need to specify a resources array. This array will take objects with the following properties:

• repo - This is the name of the repo you want to download, <username>/<repo-name>
• type - Here the type of the resource needs to be specified, for HuggingFace resources it's HF
• target - Lastly we need to define the path where the resources will be downloaded to

Example

Create a file on your filesystem called hf.json with the following contents.

{
  "version": "0.1",
  "meta": {
    "trigger": "cli"
  },
  "type": "container",
  "ops": [
    {
      "id": "huggingface",
      "args": {
        "cmd": ["ls", "/data-models"],
        "gpu": true,
        "image": "ubuntu",
        "resources": [
          {
            "type": "HF",
            "repo": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
            "target": "/data-models/"
          }
        ]
      },
      "type": "container/run"
    }
  ]
}

Deploying the Job

Now, deploy the Nosana Job Definition with the following command:

npx @nosana/cli job post --file hf.json --market nvidia-3090 --timeout 10

In this command, we specify that we want to deploy our job to the NVIDIA-3090 market, and that we want it to run for 10 minutes.

Conclusion

By following these steps, you can efficiently load resources into your Nosana deployment, leveraging the power of HuggingFace and S3. For more information, explore additional resources on Nosana documentation and HuggingFace.