Provider Routing

Provider Routing Overview

A4F integrates with multiple underlying AI providers (including Azure, AWS, Vertex AI, and select third-party services) to offer a resilient and diverse range of models. Effective routing is key to leveraging this multi-provider architecture.

You must specify a provider prefix in the model field of your API request to direct it to a particular underlying provider through A4F for Chat Completions.

How Provider Routing Works in A4F

At A4F, provider routing is achieved by prepending a specific provider prefix to the model name in your API request. The format is:

Where provider-X corresponds to one of A4F's integrated providers (e.g., provider-1, provider-3, etc.), and actual-model-identifier is the specific model ID as listed on our Models page for that provider.

If you omit the provider prefix and only specify a base model name (e.g., "claude-3-haiku-20240307"), A4F cannot determine the intended underlying source and will return an error. You must specify a provider prefix to route your request.

Understanding A4F Providers

A4F partners with multiple underlying LLM providers to offer a diverse and resilient service. Each A4F provider (provider-1 through provider-5) has distinct characteristics. Understanding these can help you make informed routing decisions:

Note: "Unlimited Context" refers to the original context window specified by the model's creator. "Limited Context" means A4F may restrict the available context window for operational reasons. Always refer to the Models page for precise model availability and naming per provider.

A4F's Strategy for Provider Integration

A4F does not perform automatic load balancing or provider selection if you omit a provider prefix; such requests will result in an error. Our strategy focuses on integrating with a diverse set of reliable underlying providers (Azure, AWS, Vertex AI, select third parties). This approach offers you:

• Choice and Control: You explicitly choose the provider using a prefix (e.g., provider-1/chatgpt-4o-latest) based on your specific needs for speed, stability, features (like function calling), or infrastructure preference.
• Resilience by Design: By partnering with multiple providers, A4F ensures that if one underlying service has issues, you can manually (in your application logic) or by pre-configuration switch to an alternative A4F provider prefix for the same or a similar model.
• Competitive Offerings: We continuously evaluate and integrate providers to offer competitive pricing and access to a wide array of models.

Provider Sorting

A4F does not offer a user-configurable sort field in the API request for provider preferences (like "price", "throughput", or "latency").

Instead, you achieve a similar outcome by selecting a specific provider prefix based on their known characteristics:

• For Speed: Choose provider-1/model-name or provider-2/model-name.
• For Stability: Choose provider-5/model-name.
• For Specific Features (e.g., Function Calling): Choose provider-3/model-name.
• For Direct Cloud Infrastructure (Azure/AWS): Choose provider-4/model-name (be mindful of potential instability).

When you explicitly specify a provider (e.g., "provider-1/chatgpt-4o-latest"), your request is routed directly to that provider for the specified model.

Ordering Specific Providers

In A4F, you "order" or prioritize a specific provider by including its prefix in the model field of your request. For example, to use gpt-4o specifically from Provider 1, you would set:

A4F does not support an order array within a provider object. If you need to implement a custom fallback chain across specific A4F providers (e.g., "try Provider 1, then Provider 3, then Provider 5"), you would need to implement this logic in your application code by making sequential requests with different provider prefixes if earlier requests fail.

Example of requesting a model via A4F using various methods:

from openai import OpenAI

A4F_API_KEY = "A4F_API_KEY"
A4F_BASE_URL = "https://api.a4f.co/v1"

client = OpenAI(
    api_key=A4F_API_KEY,
    base_url=A4F_BASE_URL,
)

try:
    completion = client.chat.completions.create(
      model="provider-1/chatgpt-4o-latest",
      messages=[
        {"role": "user", "content": "Tell me about A4F's provider routing with the OpenAI SDK."}
      ]
    )
    print(completion.choices[0].message.content)
except Exception as e:
    print(f"An error occurred: {e}")

Requiring Providers to Support All Parameters

A4F does not have a specific require_parameters field. If you need a model that supports a specific capability, such as function calling, you must route your request to the A4F provider known to support it using the correct prefix.

For example, if you need function calling capabilities (supported by provider-3):

If you send a request with parameters not supported by the explicitly chosen provider, the request may fail or the unsupported parameters may be ignored by that provider. It's best to consult the model documentation on the Models page and the known capabilities of each A4F provider.

Requiring Providers to Comply with Data Policies

A4F does not offer a specific data_collection: "deny" parameter for API requests. Our general policy is to act as a pass-through for your data to the selected underlying provider.

A4F itself does not train on your API prompt or completion data. However, the underlying providers you route to (e.g., OpenAI, Anthropic, Google via A4F's provider prefixes) have their own data usage and retention policies. You are responsible for understanding and complying with the terms of the end-providers.

Fallback Behavior

If you specify a provider prefix (e.g., provider-1/chatgpt-4o-latest) and that specific provider/model combination is unavailable or encounters an error, A4F will not automatically fall back to another provider for that request. The request will likely fail, returning an error from the targeted provider or an A4F routing error.

To implement custom fallback logic (e.g., "try Provider 1 with gpt-4o, if it fails, try Provider 2 with claude-3-haiku"), you must build this into your application by making sequential API calls to A4F with different provider-prefixed model strings.

If you do not specify a provider prefix, the request will result in an error as A4F requires explicit provider selection via a prefix.

Allowing Only Specific Providers

To ensure your request only uses a specific provider, you explicitly set the provider prefix in the model field. For example, to use only Provider 4 for a request to gpt-4o:

Ignoring Providers

A4F does not support an ignore field in the API request to explicitly exclude certain providers.

If you wish to avoid specific A4F providers, you should explicitly route your requests only to your preferred A4F providers using their prefixes (e.g., provider-1/chatgpt-4o-latest, provider-3/claude-3-haiku-20240307).

Model Availability and Naming

It's crucial to remember that model availability and naming conventions can vary between providers. A model named claude-3-opus on one A4F provider might have a slightly different identifier or may not be available on another.

Always consult the A4F Models page for the definitive list of available models and their correct identifiers for each provider prefix (e.g., provider-1/chatgpt-4o-latest, provider-2/claude-3-haiku-20240307).

Terms of Service

You can view the terms of service for A4F and our key underlying providers below. By using A4F, you agree not to violate the terms of service of third-party providers that power the models on A4F.

• A4F: https://a4f.co/terms
• AI21: https://www.ai21.com/terms-of-service
• Amazon Bedrock: https://aws.amazon.com/service-terms/
• Anthropic: https://www.anthropic.com/legal/commercial-terms
• Azure OpenAI: https://azure.microsoft.com/en-us/support/legal/cognitive-services-terms/
• Google Vertex AI: https://cloud.google.com/terms
• Mistral AI: https://mistral.ai/terms-of-use/
• OpenAI: https://openai.com/policies/terms-of-use

Note: Links to third-party terms are provided for convenience and may change. Always refer to the official provider websites for the most current terms.