How to Set Up Token Pool

The Token Pool feature multiplies your sync capacity from 600 to 3600 API calls per 10 minutes by load-balancing across multiple Vimeo API tokens. This guide shows you how to set it up.

What You'll Learn

What the Token Pool is and how it provides 6x capacity
How to generate additional Vimeo API tokens
How to configure the Token Pool
How to verify it's working correctly
How to monitor token health and usage

Prerequisites

Self-Hosted Deployment: Token Pool configuration requires access to the backend server environment variables (not available on cloud-hosted instances)
Multiple Vimeo Accounts or Tokens: You'll need 4-6 Vimeo API tokens (can be from the same account or different accounts)
Server Access: Ability to edit .env files and restart the application

Understanding Token Pool

The Problem: Rate Limits

Vimeo's API enforces rate limits to prevent abuse:

Standard Limit: 600 API calls per 10 minutes per token
What This Means: Syncing 100 videos takes ~10 minutes (1-2 API calls per video)
Large Showcases (500+ videos): Can take 60+ minutes due to rate limiting

The Solution: Token Pool

The Token Pool distributes API calls across multiple tokens:

6 Tokens: 6 × 600 = 3600 calls per 10 minutes
6x Faster: 60-minute sync completes in 10 minutes
Automatic Failover: If one token hits its rate limit, the pool switches to the next token
Zero Downtime: System continues operating even if multiple tokens are rate-limited

How It Works

Request 1 → Token 1
Request 2 → Token 2
Request 3 → Token 3
...
Request 7 → Token 1 (round-robin restarts)

If Token 1 hits rate limit:

Request 1 → Token 1 (rate limited!) → Token 2 (failover)
Token 1 → COOLDOWN (10 minutes) → auto-recovery

Step 1: Generate Vimeo API Tokens

You'll need 4-6 Vimeo API tokens. Here are two methods:

Method 1: Multiple Accounts (Recommended)

Advantages: Each account has independent rate limits (true 6x capacity)

Steps:

Create or use 6 different Vimeo accounts (free accounts work)
For each account:
- Go to Vimeo Developer Apps
- Click "Create App"
- Fill in app details (name, description, URL)
- Once created, go to "Authentication" tab
- Generate a Personal Access Token
- Select scopes: public, private, video_files
- Copy the token (you won't see it again!)
Repeat for all 6 accounts

Result: You'll have 6 tokens, each with independent 600 calls/10min rate limits.

Method 2: Single Account with Multiple Apps

Advantages: Only need one Vimeo account

Disadvantages: Rate limits may be shared across apps (not true 6x capacity)

Steps:

Log into your Vimeo account
Go to Vimeo Developer Apps
Create 6 separate apps (e.g., "Connector Token 1", "Connector Token 2", ...)
For each app, generate a Personal Access Token with scopes: public, private, video_files
Copy all 6 tokens

Important: Vimeo may still apply rate limits at the account level, so this method might not provide full 6x capacity.

Verify Your Tokens

Before proceeding, verify each token works:

curl -H "Authorization: Bearer YOUR_TOKEN_HERE" https://api.vimeo.com/me

Expected Response:

{
  "uri": "/users/123456789",
  "name": "Your Name",
  ...
}

If you see 401 Unauthorized, the token is invalid. Regenerate it.

Step 2: Create Token Configuration File

On your backend server, create a file named .env.tokens.json in the project root directory:

Location: /path/to/vimeo-connector/.env.tokens.json

Content:

{
  "tokens": [
    {
      "identifier": "token1@example.com",
      "access_token": "ab1302438decaea698520c1140c2447f",
      "scopes": ["public", "private", "video_files"]
    },
    {
      "identifier": "token2@example.com",
      "access_token": "85a7ad3303297e504c7c18919574d4d2",
      "scopes": ["public", "private", "video_files"]
    },
    {
      "identifier": "token3@example.com",
      "access_token": "c9e7f2d1a4b3e8f5d6c2a1b0e9f8d7c6",
      "scopes": ["public", "private", "video_files"]
    },
    {
      "identifier": "token4@example.com",
      "access_token": "f8e7d6c5b4a3e2f1d0c9b8a7e6f5d4c3",
      "scopes": ["public", "private", "video_files"]
    },
    {
      "identifier": "token5@example.com",
      "access_token": "a1b2c3d4e5f67890abcdef1234567890",
      "scopes": ["public", "private", "video_files"]
    },
    {
      "identifier": "token6@example.com",
      "access_token": "fedcba0987654321abcdef0123456789",
      "scopes": ["public", "private", "video_files"]
    }
  ]
}

Field Explanations:

identifier — A label for this token (use email addresses for clarity). This appears in logs and metrics.
access_token — The Vimeo Personal Access Token you generated in Step 1
scopes — Required API permissions. Always use: ["public", "private", "video_files"]

Security: This file contains secrets! Ensure:

✅ File permissions: chmod 600 .env.tokens.json (only owner can read/write)
✅ .env.tokens.json is in .gitignore (never commit to version control)
✅ Backups are encrypted

Step 3: Enable Token Pool in Environment Variables

Edit your .env file and add:

# Enable token pool
VIMEO_TOKEN_POOL_ENABLED=true

# Path to token configuration file (relative to project root)
VIMEO_TOKEN_POOL_CONFIG=.env.tokens.json

# Cooldown duration after rate limit (seconds, default: 600 = 10 min)
VIMEO_TOKEN_POOL_COOLDOWN=600

Configuration Options:

VIMEO_TOKEN_POOL_ENABLED

Values: true or false
Default: false
Description: Enables the token pool. Set to true to activate.

VIMEO_TOKEN_POOL_CONFIG

Values: File path (relative or absolute)
Default: .env.tokens.json
Description: Location of the token configuration file.

VIMEO_TOKEN_POOL_COOLDOWN

Values: Integer (seconds)
Default: 600 (10 minutes)
Description: How long a token stays in COOLDOWN state after hitting a rate limit. After this duration, the token automatically recovers and re-enters the pool.

Step 4: Restart the Application

Token Pool configuration is loaded at startup, so you must restart:

If Using Docker Compose:

docker compose down
docker compose up -d

If Running Natively:

# Stop the application (Ctrl+C or systemctl stop vimeo-connector)
# Start it again
python -m src.vimeo_connector.main

Step 5: Verify Token Pool Is Active

Check Application Logs

Look for the startup message:

docker compose logs vimeo-connector | grep -i "token pool"

Expected Output:

2025-10-17 14:22:10 [info] TokenPool initialized with 6 tokens
2025-10-17 14:22:10 [info] Token pool enabled, 6 tokens loaded

If You See:

[warning] VIMEO_TOKEN_POOL_ENABLED is true but token pool config file not found

Solution: Check that .env.tokens.json exists and the path in VIMEO_TOKEN_POOL_CONFIG is correct.

Test with a Sync Job

Start a sync job via the dashboard
Monitor the logs in real-time:

docker compose logs -f vimeo-connector

Look for token selection messages:

[info] Using token from pool token_id=toke...e.com attempt=1
[info] Using token from pool token_id=toke...e.com attempt=1
[info] Using token from pool token_id=toke...e.com attempt=1

What You Should See:

Different token_id values rotating (indicating round-robin selection)
No 429 rate limit errors (or very few, immediately followed by failover)

What You Shouldn't See:

Always the same token_id (indicates token pool isn't rotating)
Repeated 429 errors without failover (indicates failover isn't working)

Step 6: Monitor Token Pool Health

Using Prometheus Metrics

Access Metrics: http://localhost:9091/metrics

Key Metrics to Monitor:

Token Health State

vimeo_token_pool_health_state{token_id="token1@example.com"} 2

Values:

2 = HEALTHY (actively used)
1 = COOLDOWN (temporarily skipped due to rate limit)
0 = FAILED (permanently skipped due to auth error)

What to Watch:

All tokens showing 2 (HEALTHY) = perfect
1-2 tokens showing 1 (COOLDOWN) = normal under load
Any tokens showing 0 (FAILED) = investigate immediately

Token Usage Distribution

vimeo_token_pool_selections_total{token_id="token1@example.com"} 1234

What This Shows: How many times each token has been selected

What to Watch:

All tokens should have similar counts (±20%)
If one token has 10x fewer selections, it may be failing silently

Failover Events

vimeo_token_pool_failovers_total{reason="rate_limit"} 5

What This Shows: How often the pool switches tokens due to rate limits or errors

What to Watch:

0-5 failovers/hour = normal
10+ failovers/hour = approaching capacity limits (consider adding more tokens)
50+ failovers/hour = system is overloaded

Using Grafana Dashboards

Access Grafana: http://localhost:3000 (default credentials: admin / admin)

Create a Dashboard with These Panels:

Panel 1: Token Health Overview

Query: vimeo_token_pool_health_state
Visualization: Table or Stat
Purpose: See at-a-glance which tokens are healthy, in cooldown, or failed

Panel 2: Token Usage Distribution

Query: rate(vimeo_token_pool_selections_total[5m])
Visualization: Bar Chart
Purpose: Verify load is distributed evenly across all tokens

Panel 3: Failover Rate

Query: rate(vimeo_token_pool_failovers_total[5m])
Visualization: Graph
Purpose: Track how often failovers occur (spikes indicate rate limit pressure)

Using Runtime Inspection (Advanced)

If you have access to a Python shell on the backend:

from src.vimeo_connector.http.token_pool import TokenPool
from src.vimeo_connector.core.config import get_settings

settings = get_settings()
pool = TokenPool.from_json_file(settings.vimeo_token_pool_config)

# Get statistics
stats = pool.get_stats()
print(f"Total tokens: {stats['total_tokens']}")
print(f"Healthy: {stats['healthy_tokens']}")
print(f"Cooldown: {stats['cooldown_tokens']}")
print(f"Failed: {stats['failed_tokens']}")

# Detailed token info
for token in stats['token_details']:
    print(f"{token['identifier']}: {token['health_state']}")
    if 'cooldown_seconds_remaining' in token:
        print(f"  Recovers in: {token['cooldown_seconds_remaining']}s")

Example Output:

Total tokens: 6
Healthy: 5
Cooldown: 1
Failed: 0

token1@example.com: HEALTHY
token2@example.com: COOLDOWN
  Recovers in: 347s
token3@example.com: HEALTHY
token4@example.com: HEALTHY
token5@example.com: HEALTHY
token6@example.com: HEALTHY

Troubleshooting

Problem: Token Pool Not Being Used

Symptoms:

Logs show same token repeated (no rotation)
Still hitting 600 calls/10min rate limit
No "TokenPool initialized" message in logs

Checklist:

✅ VIMEO_TOKEN_POOL_ENABLED=true in .env?
✅ .env.tokens.json exists and has valid JSON?
✅ File path in VIMEO_TOKEN_POOL_CONFIG is correct (relative to project root)?
✅ Application restarted after configuration changes?

Diagnostic Commands:

# Check env var is set
docker compose exec vimeo-connector env | grep VIMEO_TOKEN_POOL

# Verify token file exists and is readable
docker compose exec vimeo-connector cat .env.tokens.json

# Check application startup logs
docker compose logs vimeo-connector | head -50

Problem: All Tokens in Cooldown

Symptoms:

Logs show: RateLimitError: All tokens are in cooldown
Syncs fail with rate limit errors
All token health states = 1 (COOLDOWN)

Cause: You're making 3600+ API calls in less than 10 minutes (exceeding even the 6x capacity)

Solutions:

Wait 10 Minutes: Tokens auto-recover after cooldown expires
Add More Tokens: Increase capacity to 12 tokens (7200 calls/10min) or 18 tokens (10,800 calls/10min)
Reduce Request Rate: Implement delays between sync jobs or batch operations

Problem: One or More Tokens Marked FAILED

Symptoms:

Token health state = 0 (FAILED)
Logs show: Authentication failed for token token_id=...
Failover happens but failed token never recovers

Causes:

Token revoked or expired
Invalid permissions/scopes
Vimeo account suspended

Solutions:

1. Verify Token Validity:

curl -H "Authorization: Bearer YOUR_TOKEN_HERE" https://api.vimeo.com/me

2. Check Token Scopes:

Go to Vimeo Developer Apps
Click on the app for this token
Go to "Authentication" tab
Verify scopes include: public, private, video_files

3. Regenerate Token:

Generate a new token for this app
Update .env.tokens.json with the new token
Restart the application:

docker compose restart vimeo-connector

Problem: Uneven Token Distribution

Symptoms:

Some tokens have 5x more usage than others in Prometheus metrics
vimeo_token_pool_selections_total shows big differences

Cause: Round-robin continues from last position after restart, or some tokens are frequently in cooldown

Is This a Problem? Usually no. Distribution evens out over time (hours).

Solution (If Persistent):

Check if low-usage tokens are frequently in COOLDOWN or FAILED state
If healthy but underused, distribution will naturally even out—monitor for 24 hours

Problem: Performance Not Improved

Symptoms:

Token pool is active but sync times haven't improved
Still taking 60 minutes to sync 500 videos

Possible Causes:

1. Network Bottleneck:

Backend server's network bandwidth is limited
Solution: Upgrade server or move to a datacenter with better Vimeo connectivity

2. Backend Server Overload:

CPU/memory constraints preventing parallel requests
Solution: Scale up server resources or optimize backend code

3. Vimeo API Throttling:

Vimeo may apply additional throttling beyond rate limits if requests are too rapid
Solution: Implement slight delays (100-200ms) between requests

4. Single-Threaded Request Processing:

Backend not making concurrent requests (processes videos one-at-a-time)
Solution: Verify backend uses async HTTP client (HTTPX with async/await)

Diagnostic:

# Monitor request rate in Prometheus
rate(vimeo_http_requests_total[1m])

Expected: ~6 req/sec with 6 tokens. If you see less than 2 req/sec, backend isn't utilizing the pool effectively.

Best Practices

Maintain Spare Capacity

Don't Run at 100% Utilization:

With 6 tokens (3600 calls/10min), target 80% usage = 2880 calls/10min
This leaves headroom for spikes and reduces cooldown events

Rule of Thumb: Plan for 20-30% buffer capacity

Monitor Token Health Proactively

Set Up Alerts:

Alert when failed_tokens > 0 (token authentication failed)
Alert when healthy_tokens < 3 (50% of capacity lost)
Alert when rate(failovers_total[5m]) > 10 (too many failovers)

Example Prometheus Alert:

groups:
  - name: token_pool
    rules:
      - alert: TokenPoolUnhealthy
        expr: vimeo_token_pool_health_state{health_state="HEALTHY"} < 3
        for: 5m
        annotations:
          summary: "Less than 3 healthy tokens in pool"

Rotate Tokens Regularly

Why: Tokens can expire or be revoked without warning

How:

Generate new tokens every 90 days
Update .env.tokens.json one token at a time
Restart application after each update
Verify old token is replaced before removing it

Document Token Sources

Maintain a Spreadsheet:

Identifier	Source Account	Created Date	Scopes	Notes
token1@...	hello@customgpt.ai	2025-01-15	public, private, video_files	Primary token
token2@...	admin@pollthepeople.app	2025-01-16	public, private, video_files	Backup token

Why: Helps track which accounts tokens come from and when they need rotation

Performance Expectations

Throughput Comparison

Configuration	Rate Limit	Expected Sync Time (500 videos)
Single Token	600 calls/10min	50-60 minutes
Token Pool (6 tokens)	3600 calls/10min	8-12 minutes
Token Pool (12 tokens)	7200 calls/10min	4-6 minutes

Variables That Affect Performance:

Transcript length (long transcripts take longer to download)
Network latency to Vimeo's API
Backend server resources (CPU, memory)
Number of concurrent sync jobs

Real-World Load Test Results

Test Setup: 6 tokens, 500 videos with transcripts, Docker Compose on 4-core server

Results:

Throughput: 5.8 req/sec sustained
Total Duration: 9 minutes 42 seconds
Rate Limit Errors: 0
Failed Requests: 2 (network timeouts, retried successfully)
Average Token Utilization: 83%

Security Considerations

Protect Token Secrets

File Permissions:

chmod 600 .env.tokens.json

Only the owner (application user) can read/write.

Never Commit to Git:

# Verify .env.tokens.json is in .gitignore
git check-ignore .env.tokens.json

Expected output: .env.tokens.json

Use Secret Management in Production:

AWS Secrets Manager
HashiCorp Vault
Kubernetes Secrets
Azure Key Vault

Example (AWS Secrets Manager):

# Store tokens in AWS Secrets Manager
aws secretsmanager create-secret \
  --name vimeo-connector/tokens \
  --secret-string file://.env.tokens.json

# Fetch at runtime in backend
import boto3
client = boto3.client('secretsmanager')
response = client.get_secret_value(SecretId='vimeo-connector/tokens')
tokens = json.loads(response['SecretString'])

Token Rotation

Recommended Schedule: Every 90 days

Rotation Process:

Generate 6 new tokens
Add new tokens to .env.tokens.json (now you have 12 tokens)
Restart application
Verify new tokens are being used (check Prometheus metrics)
Remove old tokens from .env.tokens.json
Restart application again

Zero-Downtime Rotation: Having 12 tokens temporarily ensures no capacity loss during rotation.

Limitations and Caveats

Vimeo Terms of Service

POC/Demo Use: Multiple personal tokens are acceptable for testing and demonstrations.

Production Use: May violate Vimeo's ToS. Consider:

Implementing OAuth for customer tokens
Partnering with Vimeo for official integration
Requesting higher rate limits for your app

Mitigation: This connector is designed for CustomGPT.ai's internal use with explicit permission. For third-party deployments, consult Vimeo's API team.

If Using Multiple Apps from Same Account: Vimeo may apply account-level rate limits, reducing effectiveness.

Solution: Use tokens from different Vimeo accounts (truly independent rate limits).

Manual Recovery for Failed Tokens

COOLDOWN Tokens: Recover automatically after 10 minutes

FAILED Tokens: Require manual intervention (regenerate token, update config, restart)

No Hot-Reload: Changing .env.tokens.json requires application restart (hot-reload planned for future version)

Monitor Sync Performance:

How to Monitor Sync Progress

Understand the System:

How It Works

Troubleshoot Issues:

Troubleshooting Sync Failures

Metrics and Observability:

Access Prometheus metrics at localhost:9091/metrics
See the Token Pool monitoring section above for key metrics

You're now ready to handle large-scale Vimeo syncs with 6x capacity!

What You'll Learn​

Prerequisites​

Understanding Token Pool​

The Problem: Rate Limits​

The Solution: Token Pool​

How It Works​

Step 1: Generate Vimeo API Tokens​

Method 1: Multiple Accounts (Recommended)​

Method 2: Single Account with Multiple Apps​

Verify Your Tokens​

Step 2: Create Token Configuration File​

Step 3: Enable Token Pool in Environment Variables​

Step 4: Restart the Application​

Step 5: Verify Token Pool Is Active​

Check Application Logs​

Test with a Sync Job​

Step 6: Monitor Token Pool Health​

Using Prometheus Metrics​

Token Health State​

Token Usage Distribution​

Failover Events​

Using Grafana Dashboards​

Panel 1: Token Health Overview​

Panel 2: Token Usage Distribution​

Panel 3: Failover Rate​

Using Runtime Inspection (Advanced)​

Troubleshooting​

Problem: Token Pool Not Being Used​

Problem: All Tokens in Cooldown​

Problem: One or More Tokens Marked FAILED​

Problem: Uneven Token Distribution​

Problem: Performance Not Improved​

Best Practices​

Maintain Spare Capacity​

Monitor Token Health Proactively​

Rotate Tokens Regularly​

Document Token Sources​

Performance Expectations​

Throughput Comparison​

Real-World Load Test Results​

Security Considerations​

Protect Token Secrets​

Token Rotation​

Limitations and Caveats​

Vimeo Terms of Service​

Rate Limit Sharing​

Manual Recovery for Failed Tokens​

Related Guides​

What You'll Learn

Prerequisites

Understanding Token Pool

The Problem: Rate Limits

The Solution: Token Pool

How It Works

Step 1: Generate Vimeo API Tokens

Method 1: Multiple Accounts (Recommended)

Method 2: Single Account with Multiple Apps

Verify Your Tokens

Step 2: Create Token Configuration File

Step 3: Enable Token Pool in Environment Variables

Step 4: Restart the Application

Step 5: Verify Token Pool Is Active

Check Application Logs

Test with a Sync Job

Step 6: Monitor Token Pool Health

Using Prometheus Metrics

Token Health State

Token Usage Distribution

Failover Events

Using Grafana Dashboards

Panel 1: Token Health Overview

Panel 2: Token Usage Distribution

Panel 3: Failover Rate

Using Runtime Inspection (Advanced)

Troubleshooting

Problem: Token Pool Not Being Used

Problem: All Tokens in Cooldown

Problem: One or More Tokens Marked FAILED

Problem: Uneven Token Distribution

Problem: Performance Not Improved

Best Practices

Maintain Spare Capacity

Monitor Token Health Proactively

Rotate Tokens Regularly

Document Token Sources

Performance Expectations

Throughput Comparison

Real-World Load Test Results

Security Considerations

Protect Token Secrets

Token Rotation

Limitations and Caveats

Vimeo Terms of Service

Rate Limit Sharing

Manual Recovery for Failed Tokens

Related Guides