The architectural design of a robust and secure open platform will inevitably impose rate limits on the API interfaces open to the outside world to ensure the availability of the overall system. OpenAI’s external API is no exception. We can simply download it from the official Discover API usage limits. 
[Limitations on API Doc]
[Rate limit in personal account and current level]
Limitation method
Rate limits are measured in five ways:
Requests per minute (RPM, requests per minute)
Number of requests per day (RPD, requests per day)
Tokens per minute (TPM, tokens per minute)
Number of tokens per day (TPD, tokens per day)
Images per minute (IPM, images per minute)
Rate limiting may be triggered based on which condition is reached first. For example, you might be sending 20 requests to the ChatCompletions endpoint, but only have 100 tokens, which would hit your limit (if your RPM is 20), even though 150,000 tokens are not sent in those 20 requests ( If your TPM limit is 150,000).
In practical applications, RPM is often used together with API or service limits to ensure that the system is not overloaded by excessive requests. For example, if an API has an RPM limit of 100, then the total number of requests to that API may not exceed 100 in any given minute.
It is important to note that for a more accurate RPM calculation, the actual clock time is usually used, not just the time interval from the first request to the last request. This is to ensure that the request rate per minute is accurately calculated even if the requests are unevenly distributed.
Improve usability
When developing applications using OpenAI's GPT API, you can consider the following methods to improve system availability and performance when faced with request limitations:
Use caching: Caching is an effective way to reduce the number of requests to the GPT API. For the same or similar input, you can cache the corresponding output and directly return the cached result the next time you encounter the same input without actually calling the API.
Batch requests: Consider combining multiple user requests into a single batch request. This reduces the overhead of each request and improves efficiency. However, it is important to note that merging requests may result in an increase in response time, so there is a trade-off.
Asynchronous requests: Separate user requests and API calls so that they occur asynchronously. User requests can first receive a fast response, while background asynchronous tasks are responsible for calling the GPT API and processing the results. This can reduce user waiting time.
Implement local caching: For some general or static requests, you may consider implementing local caching on the backend of your application to avoid frequent calls to the GPT API. This reduces dependence on APIs and makes your application more responsive.
Optimize input data: Ensure that the input data sent to the GPT API is minimal and necessary. By properly processing and cropping the input, the size and processing time of the request can be reduced.
Error handling and retry strategies: Implement good error handling and retry strategies to handle request failures due to network issues or API limitations. These situations can be handled efficiently using the exponential backoff retry strategy mentioned earlier.
Use multiple API Keys wisely: If your application allows it, you can use multiple OpenAI API Keys to increase the concurrency of requests. Ensure proper polling using different keys to prevent limitations of a single key from impacting overall performance.
Regular monitoring and tuning: Regularly monitor system performance and OpenAI API usage. Based on monitoring results, flexibly adjust system policies to respond to changing request patterns and API usage.
Taking these factors into consideration, you can effectively improve system availability, reduce dependence on GPT API, and provide a better user experience.
Proper use of API Key
When using the OpenAI GPT API or similar services, users are typically assigned one or more API Keys, each of which has its own request limits. By cleverly managing these API Keys, you can improve the performance and availability of your system.
Here are some specific steps and suggestions:
Obtaining multiple API Keys: If your application supports multiple API Keys, make sure you obtain multiple valid API Keys. You can create a new API Key on the OpenAI console.
Polling using a different API Key: In your application code, implement a mechanism to poll using a different API Key. This ensures that each API Key has a chance to be used, preventing a single key from reaching the request limit and causing overall performance degradation.
Switch API Key on error: When an error is encountered when sending a request using one API Key (for example, the request limit is reached), immediately switch to another API Key and try again. This can be an automated process to ensure that applications can quickly switch to other available keys in the event of an error.
Monitor API Key usage: Regularly monitor the usage of each API Key to understand the request frequency and success rate of each key. This can help you determine if you need to change the order in which keys are used or if you need to adjust the requested distribution policy.
Balancing concurrency and request limits: Although using multiple API keys can improve concurrency, be careful not to exceed the total request limit of the OpenAI API. Make sure your system can stay within the total number of requests allowed when using multiple keys.
Security considerations: Ensure the security of the API Key. Avoid hardcoding sensitive information in your application code and take necessary security measures, such as using environment variables or dedicated secure storage to save API keys.
Through these methods, you can maximize the use of multiple API Keys, improve system concurrency and performance, and ensure that the OpenAI GPT API can still be used effectively under high request loads.
—Extended reading—
WPS Office AI practical summary, the era of intelligent office has arrived
This is the best translation software I have ever used, don’t miss it
Breaking the 35-year-old mid-life crisis
A book that can affect your (child's) life, not to be missed
A big AI model with a big name but no useful purpose. It doesn’t live up to its name.