« Evaluating Narwhal
Chat/document workload results
| Field | Value |
|---|
| Evaluation dates | 14 to 17 September 2026 |
| Workload | Five chat/document/chat cycles |
| Model | moonshotai/Kimi-K3 |
| Fleet | Six TP8 engines |
| Frameworks | Narwhal; Dynamo Planner; Ray Serve LLM |
| Prefix-caching modes | Off; On |
| Offers per configuration | 14,250 |
| Chat offers per configuration | 6,810 |
| Document offers per configuration | 7,440 |
| TTFT SLO | At or below 2.1 seconds |
| TPOT SLO | At or below 45 milliseconds |
Serving versions and settings are on the serving configurations page.
Request outcomes and SLO delivery
| Framework | Prefix caching | Offered | Completed | Failed | Refused | TTFT SLO delivered | TPOT SLO delivered | Joint latency SLO delivered | SLO-qualified |
|---|
| Dynamo Planner | Off | 14,250 | 6,973 (48.93%) | 9 (0.06%) | 7,268 (51.00%) | 4,033 (28.30%) | 6,982 (49.00%) | 4,033 (28.30%) | 3,812 (26.75%) |
| Dynamo Planner | On | 14,250 | 9,380 (65.82%) | 47 (0.33%) | 4,823 (33.85%) | 8,303 (58.27%) | 9,427 (66.15%) | 8,303 (58.27%) | 8,088 (56.76%) |
| Narwhal | Off | 14,250 | 13,044 (91.54%) | 807 (5.66%) | 399 (2.80%) | 6,143 (43.11%) | 12,984 (91.12%) | 6,072 (42.61%) | 5,743 (40.30%) |
| Narwhal | On | 14,250 | 13,074 (91.75%) | 765 (5.37%) | 411 (2.88%) | 11,196 (78.57%) | 13,065 (91.68%) | 11,187 (78.51%) | 10,855 (76.18%) |
| Ray Serve LLM | Off | 14,250 | 7,731 (54.25%) | 6,519 (45.75%) | 0 (0.00%) | 2,409 (16.91%) | 7,731 (54.25%) | 2,409 (16.91%) | 2,262 (15.87%) |
| Ray Serve LLM | On | 14,250 | 10,059 (70.59%) | 4,191 (29.41%) | 0 (0.00%) | 8,633 (60.58%) | 8,489 (59.57%) | 7,502 (52.65%) | 7,361 (51.66%) |
Completion termination
| Framework | Prefix caching | Natural completions | Output-cap completions |
|---|
| Dynamo Planner | Off | 6,674 (95.71%) | 299 (4.29%) |
| Dynamo Planner | On | 9,142 (97.46%) | 238 (2.54%) |
| Narwhal | Off | 12,608 (96.66%) | 436 (3.34%) |
| Narwhal | On | 12,615 (96.49%) | 459 (3.51%) |
| Ray Serve LLM | Off | 7,465 (96.56%) | 266 (3.44%) |
| Ray Serve LLM | On | 9,860 (98.02%) | 199 (1.98%) |
Document answer quality
| Framework | Prefix caching | Scored document offers | All-offer token F1 | Completed-only token F1 | SLO-delivered token F1 |
|---|
| Dynamo Planner | Off | 7,440 | 21.1205% | 48.0687% | 9.8410% |
| Dynamo Planner | On | 7,440 | 40.0627% | 48.6639% | 39.0870% |
| Narwhal | Off | 7,440 | 48.3067% | 48.7193% | 12.0275% |
| Narwhal | On | 7,440 | 48.4097% | 48.8364% | 45.4206% |
| Ray Serve LLM | Off | 7,440 | 24.8693% | 48.2220% | 0.6535% |
| Ray Serve LLM | On | 7,440 | 43.0827% | 48.3535% | 36.4542% |
Completed-request latency
| Framework | Prefix caching | TTFT p50 (s) | TTFT p95 (s) | TTFT p99 (s) | TPOT p50 (ms) | TPOT p95 (ms) | TPOT p99 (ms) |
|---|
| Dynamo Planner | Off | 1.796 | 9.116 | 15.263 | 32.473 | 36.019 | 36.811 |
| Dynamo Planner | On | 0.627 | 7.943 | 14.288 | 31.801 | 36.129 | 36.795 |
| Narwhal | Off | 2.290 | 11.684 | 17.653 | 32.654 | 35.985 | 36.327 |
| Narwhal | On | 0.631 | 3.601 | 14.096 | 34.523 | 36.174 | 36.736 |
| Ray Serve LLM | Off | 5.203 | 30.527 | 33.490 | 26.765 | 29.665 | 29.773 |
| Ray Serve LLM | On | 0.676 | 4.112 | 6.833 | 27.729 | 51.602 | 59.880 |
Completed-request E2E latency
| Framework | Prefix caching | E2E p50 (s) | E2E p95 (s) | E2E p99 (s) |
|---|
| Dynamo Planner | Off | 12.834 | 238.129 | 289.857 |
| Dynamo Planner | On | 7.294 | 168.233 | 282.128 |
| Narwhal | Off | 12.698 | 216.603 | 293.741 |
| Narwhal | On | 8.717 | 213.885 | 292.877 |
| Ray Serve LLM | Off | 30.030 | 184.585 | 241.990 |
| Ray Serve LLM | On | 6.944 | 146.450 | 243.057 |
Outcomes by request class
| Framework | Prefix caching | Request class | Offered | Completed | Failed | Refused | SLO-qualified |
|---|
| Dynamo Planner | Off | Chat | 6,810 | 3,704 (54.39%) | 9 (0.13%) | 3,097 (45.48%) | 2,307 (33.88%) |
| Dynamo Planner | Off | Document | 7,440 | 3,269 (43.94%) | 0 (0.00%) | 4,171 (56.06%) | 1,505 (20.23%) |
| Dynamo Planner | On | Chat | 6,810 | 3,255 (47.80%) | 45 (0.66%) | 3,510 (51.54%) | 2,118 (31.10%) |
| Dynamo Planner | On | Document | 7,440 | 6,125 (82.33%) | 2 (0.03%) | 1,313 (17.65%) | 5,970 (80.24%) |
| Narwhal | Off | Chat | 6,810 | 5,667 (83.22%) | 778 (11.42%) | 365 (5.36%) | 3,985 (58.52%) |
| Narwhal | Off | Document | 7,440 | 7,377 (99.15%) | 29 (0.39%) | 34 (0.46%) | 1,758 (23.63%) |
| Narwhal | On | Chat | 6,810 | 5,699 (83.69%) | 737 (10.82%) | 374 (5.49%) | 3,961 (58.16%) |
| Narwhal | On | Document | 7,440 | 7,375 (99.13%) | 28 (0.38%) | 37 (0.50%) | 6,894 (92.66%) |
| Ray Serve LLM | Off | Chat | 6,810 | 3,894 (57.18%) | 2,916 (42.82%) | 0 (0.00%) | 2,163 (31.76%) |
| Ray Serve LLM | Off | Document | 7,440 | 3,837 (51.57%) | 3,603 (48.43%) | 0 (0.00%) | 99 (1.33%) |
| Ray Serve LLM | On | Chat | 6,810 | 3,430 (50.37%) | 3,380 (49.63%) | 0 (0.00%) | 1,749 (25.68%) |
| Ray Serve LLM | On | Document | 7,440 | 6,629 (89.10%) | 811 (10.90%) | 0 (0.00%) | 5,612 (75.43%) |
Failed-request classification
| Framework | Prefix caching | Request class | Failed | Before first output token | After output started | Exact cause classified | Exact cause unavailable |
|---|
| Dynamo Planner | Off | Chat | 9 | 0 | 9 | 0 | 9 |
| Dynamo Planner | Off | Document | 0 | 0 | 0 | 0 | 0 |
| Dynamo Planner | On | Chat | 45 | 0 | 45 | 0 | 45 |
| Dynamo Planner | On | Document | 2 | 0 | 2 | 0 | 2 |
| Narwhal | Off | Chat | 778 | 731 | 47 | 778 | 0 |
| Narwhal | Off | Document | 29 | 29 | 0 | 29 | 0 |
| Narwhal | On | Chat | 737 | 737 | 0 | 737 | 0 |
| Narwhal | On | Document | 28 | 28 | 0 | 28 | 0 |
| Ray Serve LLM | Off | Chat | 2,916 | 2,916 | 0 | 2,916 | 0 |
| Ray Serve LLM | Off | Document | 3,603 | 3,603 | 0 | 3,603 | 0 |
| Ray Serve LLM | On | Chat | 3,380 | 3,319 | 61 | 3,380 | 0 |
| Ray Serve LLM | On | Document | 811 | 811 | 0 | 811 | 0 |
Exact failure causes
| Framework | Prefix caching | Request class | Cause | Count | Max tokens |
|---|
| Narwhal | Off | Chat | original_request_deadline_expired | 47 | 8192 |
| Narwhal | Off | Chat | queue_expired | 731 | 8192 |
| Narwhal | Off | Document | queue_expired | 29 | 8192 |
| Narwhal | On | Chat | queue_expired | 737 | 8192 |
| Narwhal | On | Document | queue_expired | 28 | 8192 |
| Ray Serve LLM | Off | Chat | service_unavailable | 2,916 | 8192 |
| Ray Serve LLM | Off | Document | service_unavailable | 3,603 | 8192 |
| Ray Serve LLM | On | Chat | clientpayloaderror | 61 | 8192 |
| Ray Serve LLM | On | Chat | service_unavailable | 3,319 | 8192 |
| Ray Serve LLM | On | Document | service_unavailable | 811 | 8192 |
| Framework | Prefix caching | HTTP status | Error type | Error message | Count |
|---|
| Narwhal | Off | 200 | decode | RequestExpired: original request deadline expired | 47 |
| Narwhal | Off | 504 | Gateway Timeout | queue deadline expired | 760 |
| Narwhal | On | 504 | Gateway Timeout | queue deadline expired | 765 |
| Ray Serve LLM | Off | 503 | Service Unavailable | Request dropped due to backpressure (num_queued_requests=4, max_queued_requests=4). | 6,424 |
| Ray Serve LLM | Off | 503 | Service Unavailable | Request dropped due to backpressure (num_queued_requests=5, max_queued_requests=4). | 79 |
| Ray Serve LLM | Off | 503 | Service Unavailable | Request dropped due to backpressure (num_queued_requests=6, max_queued_requests=4). | 4 |
| Ray Serve LLM | Off | 503 | Service Unavailable | Request dropped due to backpressure (num_queued_requests=7, max_queued_requests=4). | 8 |
| Ray Serve LLM | Off | 503 | Service Unavailable | Request dropped due to backpressure (num_queued_requests=8, max_queued_requests=4). | 4 |
| Ray Serve LLM | On | 200 | ClientPayloadError | ClientPayloadError("Response payload is not completed: <TransferEncodingError: 400, message='Not enough data to satisfy transfer length header.'>") | 61 |
| Ray Serve LLM | On | 503 | Service Unavailable | Request dropped due to backpressure (num_queued_requests=4, max_queued_requests=4). | 4,123 |
| Ray Serve LLM | On | 503 | Service Unavailable | Request dropped due to backpressure (num_queued_requests=5, max_queued_requests=4). | 7 |
Queue telemetry
| Framework | Prefix caching | Queue wait p50 (s) | Queue wait p95 (s) | Queue wait p99 (s) | Queued requests p50 | Queued requests p95 | Queued requests peak |
|---|
| Narwhal | Off | 0.800 | 9.674 | 14.294 | 0.00 | 3.00 | 4.00 |
| Narwhal | On | 0.001 | 3.258 | 11.848 | 0.00 | 3.00 | 4.00 |
Definitions
-
Percentages for completed, failed, refused and SLO delivery use all offered requests as the denominator.
-
Natural-completion and output-cap percentages use completed requests as the denominator.
-
All-offer token F1 averages normalized token-overlap F1 over all 7,440 document offers. Failed and refused document offers contribute zero.
-
Completed-only token F1 is the mean over completed document offers.
-
SLO-delivered token F1 sums F1 from SLO-qualified document offers and divides by all 7,440 document offers.
-
Chat requests are not scored for F1.
-
An SLO-qualified request completed naturally with a nonempty answer and at least two observed tokens. Its TTFT was at or below 2.1 seconds and its TPOT at or below 45 milliseconds.
-
Failure stage uses one rule for every framework: zero observed output tokens means before first output token; one or more means after output started.
-
Each cause comes from the top-level transport error or streamed error event.
Athrael.net