Deploy Inference.net nodes using Docker on Windows and Linux
This guide provides detailed instructions for deploying Inference.net nodes using Docker. If you’re new to Docker, please refer to the Docker documentation. Join our Discord community if you need assistance.
Enter a worker name, ensure Docker is selected, and click Create Worker
On the Worker Details page, click Launch Worker
Run the provided Docker command with your worker code:
Copy
Ask AI
docker run \ --pull=always \ --restart=always \ --runtime=nvidia \ --gpus all \ -v ~/.inference:/root/.inference \ inferencedevnet/amd64-nvidia-inference-node:latest \ --code <your-worker-code>
Once started, your node will enter the “Initializing” state on the dashboard. This initialization phase typically takes 1-2 minutes but may take up to 10 minutes depending on your GPU.
Enter a worker name, ensure Docker is selected, and click Create Worker
On the Worker Details page, click Launch Worker
Run the provided Docker command with your worker code in PowerShell:
Copy
Ask AI
docker run ` --pull=always ` --restart=always ` --runtime=nvidia ` --gpus all ` -v ~/.inference:/root/.inference ` inferencedevnet/amd64-nvidia-inference-node:latest ` --code <your-worker-code>
Once started, your node will enter the “Initializing” state on the dashboard. This preparation phase typically takes 1-2 minutes but may take up to 10 minutes depending on your GPU.
If you have multiple GPUs, you can run separate containers, each utilizing a different GPU. This maximizes your hardware utilization by running multiple workers simultaneously.
GPUs are numbered starting from 0. You can specify which GPU(s) each container should use:
Copy
Ask AI
# Use all available GPUsdocker run --gpus all ...# Use specific GPUs (e.g., first and second GPU)docker run --gpus '"device=0,1"' ...# Use a single GPU (e.g., first GPU)docker run --gpus '"device=0"' ...
For easier management of multiple containers, you can use Docker Compose. This is particularly useful when running multiple GPU instances or managing complex deployments.Create a docker-compose.yml file:
# Start all servicesdocker-compose up -d# View logs for all servicesdocker-compose logs -f# Stop all servicesdocker-compose down# Restart a specific instancedocker-compose restart instance-0
When your Docker container fails to start or stops unexpectedly, these commands help diagnose and resolve Docker daemon-related issues. Use them to check if Docker is running properly, restart the service if needed, or examine system logs for error messages.
These commands help troubleshoot GPU visibility and accessibility issues within Docker containers. Use them to verify that the NVIDIA runtime is properly configured, list available GPUs on your system, or reset a GPU that may be in an error state.
Copy
Ask AI
# Verify NVIDIA runtimedocker info | grep nvidia# List available GPUsnvidia-smi -L# Reset GPU (if needed - Linux)sudo nvidia-smi --gpu-reset
Monitor GPU performance and resource utilization in real-time. These commands are essential for tracking GPU memory usage, identifying bottlenecks, and ensuring your inference workloads are running efficiently. Use them to detect memory leaks or verify that your containers are properly utilizing GPU resources.
Essential commands for managing Docker containers running your inference nodes. Use these to check container health, view logs for debugging, follow real-time output during operations, or monitor resource consumption to ensure containers aren’t exceeding system limits.
Commands for managing failed or problematic containers. Use these when you need to stop unresponsive containers, force-terminate hung processes, remove containers for a fresh start, or clean up disk space by removing unused Docker resources.
Copy
Ask AI
# Stop container gracefullydocker stop <container_name># Force stopdocker kill <container_name># Remove containerdocker rm <container_name># Clean up unused resourcesdocker system prune
Configure automatic container restart behavior to ensure high availability of your inference nodes. These policies help maintain uptime by automatically restarting containers after crashes, system reboots, or failures, reducing the need for manual intervention.
Copy
Ask AI
# Always restart (including after reboot)docker run -d --restart always ...# Restart on failure (max 5 attempts)docker run -d --restart on-failure:5 ...# Restart unless manually stoppeddocker run -d --restart unless-stopped ...
Advanced monitoring commands for tracking both container and GPU performance metrics. Use these to identify performance bottlenecks, monitor power consumption for efficiency optimization, or track temperature to prevent thermal throttling during intensive inference workloads.
Copy
Ask AI
# Monitor container metricsdocker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}"# Monitor GPU temperature and powernvidia-smi --query-gpu=temperature.gpu,power.draw,power.limit --format=csv -l 1
Tip: Always check container logs first when troubleshooting issues. They often contain valuable error messages and diagnostic information.