Measure the experience your customer sees
For streaming text, time to first token measures how long a user waits before the first output arrives. Total request duration measures how long the complete result takes. A quick first token can still be followed by a slow response. Track both, along with output length and error rate.
Measure from the regions where your application runs and where customers connect. Network latency is only part of the result: input length, model processing, load, and output length also matter. Use representative tasks and look at slower requests, not just the average.
Understand rate limits and bursts
Requests per minute (RPM) and tokens per minute (TPM) describe different limits. An application can hit a token limit before a request limit when prompts are long. Confirm how input, output, and concurrent requests count for the models on your account.
Smooth bursts with a queue and use bounded retries with backoff. Respect Retry-After when provided. Repeating a timed-out generation may incur another charge or create another video task; reconcile task state first when the endpoint supports it. Do not assume that a retry is idempotent.
Decide what is safe to send and retain
Minimize personal and confidential data in prompts. Review your gateway’s policy and the relevant upstream provider’s terms for processing, retention, training use, and geographic restrictions. Do not assume a gateway changes an upstream provider’s data policy.
Restrict access to API keys and operational logs. Keep diagnostic records sufficient for troubleshooting without logging full prompts by default. Choose retention and deletion rules that fit your contractual obligations. For sensitive workloads, obtain the necessary agreements before transmitting production data.
- Review upstream policies and geographic availability.
- Separate test credentials from production credentials.
- Define log access and retention.
- Assign an owner for incident response.
Make failures understandable
Show users a useful error state rather than an indefinite spinner. Record a request identifier, timestamp, model ID, status, and elapsed time. Avoid putting credentials or full customer content into error reports.
Prepare the support path before launch. SinoRouter offers 24/7 human support, but any guaranteed response time, availability target, or custom service arrangement must be agreed separately. Run a small rollout first, review real usage and costs, then increase traffic deliberately.
Reference and next checks
Use the applicable provider or application documentation alongside your account-specific route information. These guides explain configuration and evaluation; they do not certify live support for every feature.