« Evaluating Narwhal
Mixed workload results
| Field | Value |
|---|
| Evaluation dates | 17 to 18 September 2026 |
| Workload | Five request classes across nine consecutive load phases |
| Model | moonshotai/Kimi-K3 |
| Fleet | Six TP8 engines |
| Frameworks | Narwhal; Dynamo Planner; Ray Serve LLM |
| Prefix-caching modes | Off; On |
| Offers per configuration | 14,419 |
| TTFT SLO | At or below 2.1 seconds |
| TPOT SLO | At or below 45 milliseconds |
Serving versions and settings are on the serving configurations page.
Request outcomes and SLO delivery
| Framework | Prefix caching | Offered | Completed | Failed | Refused | TTFT SLO delivered | TPOT SLO delivered | Joint latency SLO delivered | SLO-qualified |
|---|
| Dynamo Planner | Off | 14,419 | 8,298 (57.55%) | 19 (0.13%) | 6,102 (42.32%) | 4,554 (31.58%) | 8,317 (57.68%) | 4,554 (31.58%) | 2,144 (14.87%) |
| Dynamo Planner | On | 14,419 | 8,131 (56.39%) | 11 (0.08%) | 6,277 (43.53%) | 6,027 (41.80%) | 8,142 (56.47%) | 6,027 (41.80%) | 3,135 (21.74%) |
| Narwhal | Off | 14,419 | 13,999 (97.09%) | 48 (0.33%) | 372 (2.58%) | 12,167 (84.38%) | 13,999 (97.09%) | 12,167 (84.38%) | 6,327 (43.88%) |
| Narwhal | On | 14,419 | 13,973 (96.91%) | 51 (0.35%) | 395 (2.74%) | 12,427 (86.18%) | 13,973 (96.91%) | 12,427 (86.18%) | 6,579 (45.63%) |
| Ray Serve LLM | Off | 14,419 | 12,364 (85.75%) | 2,055 (14.25%) | 0 (0.00%) | 7,943 (55.09%) | 12,362 (85.73%) | 7,943 (55.09%) | 3,670 (25.45%) |
| Ray Serve LLM | On | 14,419 | 13,569 (94.11%) | 850 (5.89%) | 0 (0.00%) | 13,462 (93.36%) | 10,825 (75.07%) | 10,767 (74.67%) | 5,353 (37.12%) |
Failure stage
| Framework | Prefix caching | Request class | Failed | Before first output token | After output started |
|---|
| Dynamo Planner | Off | Conversation History | 1 | 0 | 1 |
| Dynamo Planner | Off | Document | 0 | 0 | 0 |
| Dynamo Planner | Off | Long Generation | 0 | 0 | 0 |
| Dynamo Planner | Off | Short Chat | 9 | 0 | 9 |
| Dynamo Planner | Off | Standard Chat | 9 | 0 | 9 |
| Dynamo Planner | On | Conversation History | 0 | 0 | 0 |
| Dynamo Planner | On | Document | 0 | 0 | 0 |
| Dynamo Planner | On | Long Generation | 2 | 0 | 2 |
| Dynamo Planner | On | Short Chat | 2 | 0 | 2 |
| Dynamo Planner | On | Standard Chat | 7 | 0 | 7 |
| Narwhal | Off | Conversation History | 0 | 0 | 0 |
| Narwhal | Off | Document | 31 | 31 | 0 |
| Narwhal | Off | Long Generation | 3 | 3 | 0 |
| Narwhal | Off | Short Chat | 11 | 11 | 0 |
| Narwhal | Off | Standard Chat | 3 | 3 | 0 |
| Narwhal | On | Conversation History | 1 | 1 | 0 |
| Narwhal | On | Document | 31 | 31 | 0 |
| Narwhal | On | Long Generation | 7 | 7 | 0 |
| Narwhal | On | Short Chat | 8 | 8 | 0 |
| Narwhal | On | Standard Chat | 4 | 4 | 0 |
| Ray Serve LLM | Off | Conversation History | 34 | 34 | 0 |
| Ray Serve LLM | Off | Document | 1,295 | 1,295 | 0 |
| Ray Serve LLM | Off | Long Generation | 61 | 61 | 0 |
| Ray Serve LLM | Off | Short Chat | 482 | 482 | 0 |
| Ray Serve LLM | Off | Standard Chat | 183 | 183 | 0 |
| Ray Serve LLM | On | Conversation History | 7 | 7 | 0 |
| Ray Serve LLM | On | Document | 482 | 482 | 0 |
| Ray Serve LLM | On | Long Generation | 39 | 39 | 0 |
| Ray Serve LLM | On | Short Chat | 239 | 239 | 0 |
| Ray Serve LLM | On | Standard Chat | 83 | 83 | 0 |
Recorded terminal signatures
| Framework | Prefix caching | Outcome | HTTP status | Cause | Requests |
|---|
| Dynamo Planner | Off | Failed | 200 | 500 | 19 |
| Dynamo Planner | Off | Refused | 529 | http 529 refusal | 6,102 |
| Dynamo Planner | On | Failed | 200 | 500 | 11 |
| Dynamo Planner | On | Refused | 529 | http 529 refusal | 6,277 |
| Narwhal | Off | Failed | 504 | queue expired | 48 |
| Narwhal | Off | Refused | 429 | server overloaded error | 372 |
| Narwhal | On | Failed | 504 | queue expired | 51 |
| Narwhal | On | Refused | 429 | server overloaded error | 395 |
| Ray Serve LLM | Off | Failed | 503 | service unavailable | 2,055 |
| Ray Serve LLM | On | Failed | 503 | service unavailable | 850 |
Completion termination
| Framework | Prefix caching | Natural completions | Output-cap completions |
|---|
| Dynamo Planner | Off | 4,404 (53.07%) | 3,894 (46.93%) |
| Dynamo Planner | On | 4,267 (52.48%) | 3,864 (47.52%) |
| Narwhal | Off | 7,444 (53.18%) | 6,555 (46.82%) |
| Narwhal | On | 7,470 (53.46%) | 6,503 (46.54%) |
| Ray Serve LLM | Off | 6,443 (52.11%) | 5,921 (47.89%) |
| Ray Serve LLM | On | 7,185 (52.95%) | 6,384 (47.05%) |
Answer quality
| Framework | Prefix caching | Scored offers | All-offer token F1 | Completed-only token F1 | SLO-delivered token F1 |
|---|
| Dynamo Planner | Off | 14,419 | 17.0010% | 29.5417% | 7.0999% |
| Dynamo Planner | On | 14,419 | 16.1254% | 28.5958% | 10.6774% |
| Narwhal | Off | 14,419 | 28.5651% | 29.4221% | 21.8589% |
| Narwhal | On | 14,419 | 28.3793% | 29.2851% | 22.6357% |
| Ray Serve LLM | Off | 14,419 | 24.2343% | 28.2622% | 11.9505% |
| Ray Serve LLM | On | 14,419 | 27.5034% | 29.2263% | 18.1647% |
Completed-request latency
| Framework | Prefix caching | TTFT p50 (s) | TTFT p95 (s) | TTFT p99 (s) | TPOT p50 (ms) | TPOT p95 (ms) | TPOT p99 (ms) |
|---|
| Dynamo Planner | Off | 1.954 | 3.960 | 5.018 | 35.916 | 36.648 | 36.987 |
| Dynamo Planner | On | 1.405 | 3.520 | 4.667 | 36.040 | 36.687 | 37.091 |
| Narwhal | Off | 0.576 | 11.204 | 13.460 | 32.701 | 36.188 | 36.660 |
| Narwhal | On | 0.384 | 11.268 | 14.203 | 32.760 | 36.256 | 36.772 |
| Ray Serve LLM | Off | 0.893 | 16.898 | 18.623 | 23.508 | 26.125 | 28.425 |
| Ray Serve LLM | On | 0.489 | 1.183 | 1.917 | 26.227 | 49.747 | 50.313 |
Completed-request E2E latency
| Framework | Prefix caching | E2E p50 (s) | E2E p95 (s) | E2E p99 (s) |
|---|
| Dynamo Planner | Off | 10.584 | 60.444 | 113.011 |
| Dynamo Planner | On | 10.351 | 62.339 | 114.572 |
| Narwhal | Off | 8.742 | 55.266 | 109.099 |
| Narwhal | On | 8.450 | 54.768 | 108.823 |
| Ray Serve LLM | Off | 11.325 | 47.786 | 87.713 |
| Ray Serve LLM | On | 7.738 | 52.812 | 95.383 |
Outcomes by request class
| Framework | Prefix caching | Request class | Offered | Completed | Failed | Refused | SLO-qualified |
|---|
| Dynamo Planner | Off | Conversation History | 474 | 311 (65.61%) | 1 (0.21%) | 162 (34.18%) | 89 (18.78%) |
| Dynamo Planner | Off | Document | 6,310 | 3,539 (56.09%) | 0 (0.00%) | 2,771 (43.91%) | 1,141 (18.08%) |
| Dynamo Planner | Off | Long Generation | 550 | 252 (45.82%) | 0 (0.00%) | 298 (54.18%) | 45 (8.18%) |
| Dynamo Planner | Off | Short Chat | 4,956 | 2,887 (58.25%) | 9 (0.18%) | 2,060 (41.57%) | 490 (9.89%) |
| Dynamo Planner | Off | Standard Chat | 2,129 | 1,309 (61.48%) | 9 (0.42%) | 811 (38.09%) | 379 (17.80%) |
| Dynamo Planner | On | Conversation History | 474 | 290 (61.18%) | 0 (0.00%) | 184 (38.82%) | 85 (17.93%) |
| Dynamo Planner | On | Document | 6,310 | 3,259 (51.65%) | 0 (0.00%) | 3,051 (48.35%) | 1,966 (31.16%) |
| Dynamo Planner | On | Long Generation | 550 | 258 (46.91%) | 2 (0.36%) | 290 (52.73%) | 59 (10.73%) |
| Dynamo Planner | On | Short Chat | 4,956 | 2,934 (59.20%) | 2 (0.04%) | 2,020 (40.76%) | 579 (11.68%) |
| Dynamo Planner | On | Standard Chat | 2,129 | 1,390 (65.29%) | 7 (0.33%) | 732 (34.38%) | 446 (20.95%) |
| Narwhal | Off | Conversation History | 474 | 468 (98.73%) | 0 (0.00%) | 6 (1.27%) | 184 (38.82%) |
| Narwhal | Off | Document | 6,310 | 6,065 (96.12%) | 31 (0.49%) | 214 (3.39%) | 4,006 (63.49%) |
| Narwhal | Off | Long Generation | 550 | 531 (96.55%) | 3 (0.55%) | 16 (2.91%) | 171 (31.09%) |
| Narwhal | Off | Short Chat | 4,956 | 4,838 (97.62%) | 11 (0.22%) | 107 (2.16%) | 1,127 (22.74%) |
| Narwhal | Off | Standard Chat | 2,129 | 2,097 (98.50%) | 3 (0.14%) | 29 (1.36%) | 839 (39.41%) |
| Narwhal | On | Conversation History | 474 | 469 (98.95%) | 1 (0.21%) | 4 (0.84%) | 187 (39.45%) |
| Narwhal | On | Document | 6,310 | 6,058 (96.01%) | 31 (0.49%) | 221 (3.50%) | 4,207 (66.67%) |
| Narwhal | On | Long Generation | 550 | 526 (95.64%) | 7 (1.27%) | 17 (3.09%) | 170 (30.91%) |
| Narwhal | On | Short Chat | 4,956 | 4,830 (97.46%) | 8 (0.16%) | 118 (2.38%) | 1,167 (23.55%) |
| Narwhal | On | Standard Chat | 2,129 | 2,090 (98.17%) | 4 (0.19%) | 35 (1.64%) | 848 (39.83%) |
| Ray Serve LLM | Off | Conversation History | 474 | 440 (92.83%) | 34 (7.17%) | 0 (0.00%) | 167 (35.23%) |
| Ray Serve LLM | Off | Document | 6,310 | 5,015 (79.48%) | 1,295 (20.52%) | 0 (0.00%) | 1,774 (28.11%) |
| Ray Serve LLM | Off | Long Generation | 550 | 489 (88.91%) | 61 (11.09%) | 0 (0.00%) | 144 (26.18%) |
| Ray Serve LLM | Off | Short Chat | 4,956 | 4,474 (90.27%) | 482 (9.73%) | 0 (0.00%) | 909 (18.34%) |
| Ray Serve LLM | Off | Standard Chat | 2,129 | 1,946 (91.40%) | 183 (8.60%) | 0 (0.00%) | 676 (31.75%) |
| Ray Serve LLM | On | Conversation History | 474 | 467 (98.52%) | 7 (1.48%) | 0 (0.00%) | 179 (37.76%) |
| Ray Serve LLM | On | Document | 6,310 | 5,828 (92.36%) | 482 (7.64%) | 0 (0.00%) | 3,159 (50.06%) |
| Ray Serve LLM | On | Long Generation | 550 | 511 (92.91%) | 39 (7.09%) | 0 (0.00%) | 187 (34.00%) |
| Ray Serve LLM | On | Short Chat | 4,956 | 4,717 (95.18%) | 239 (4.82%) | 0 (0.00%) | 1,043 (21.05%) |
| Ray Serve LLM | On | Standard Chat | 2,129 | 2,046 (96.10%) | 83 (3.90%) | 0 (0.00%) | 785 (36.87%) |
Answer quality by request class
| Framework | Prefix caching | Request class | Scored offers | All-offer token F1 | Completed-only token F1 | SLO-delivered token F1 |
|---|
| Dynamo Planner | Off | Conversation History | 474 | 13.7066% | 20.8904% | 7.3886% |
| Dynamo Planner | Off | Document | 6,310 | 27.3484% | 48.7620% | 10.3557% |
| Dynamo Planner | Off | Long Generation | 550 | 8.4483% | 18.4387% | 3.3382% |
| Dynamo Planner | Off | Short Chat | 4,956 | 6.0550% | 10.3943% | 3.1610% |
| Dynamo Planner | Off | Standard Chat | 2,129 | 14.7567% | 24.0008% | 7.5268% |
| Dynamo Planner | On | Conversation History | 474 | 11.9914% | 19.5998% | 7.0334% |
| Dynamo Planner | On | Document | 6,310 | 24.9341% | 48.2768% | 17.6084% |
| Dynamo Planner | On | Long Generation | 550 | 9.0745% | 19.3448% | 3.7917% |
| Dynamo Planner | On | Short Chat | 4,956 | 6.2247% | 10.5144% | 3.7443% |
| Dynamo Planner | On | Standard Chat | 2,129 | 15.8074% | 24.2115% | 8.8645% |
| Narwhal | Off | Conversation History | 474 | 19.8877% | 20.1427% | 15.4645% |
| Narwhal | Off | Document | 6,310 | 46.3692% | 48.2423% | 36.1500% |
| Narwhal | Off | Long Generation | 550 | 18.0898% | 18.7371% | 12.0006% |
| Narwhal | Off | Short Chat | 4,956 | 10.3636% | 10.6164% | 7.6363% |
| Narwhal | Off | Standard Chat | 2,129 | 22.8053% | 23.1533% | 16.5807% |
| Narwhal | On | Conversation History | 474 | 19.9770% | 20.1900% | 15.1476% |
| Narwhal | On | Document | 6,310 | 46.0606% | 47.9766% | 37.8770% |
| Narwhal | On | Long Generation | 550 | 17.4523% | 18.2486% | 11.5444% |
| Narwhal | On | Short Chat | 4,956 | 10.3374% | 10.6071% | 7.7393% |
| Narwhal | On | Standard Chat | 2,129 | 22.6671% | 23.0901% | 16.6720% |
| Ray Serve LLM | Off | Conversation History | 474 | 18.7679% | 20.2182% | 13.5435% |
| Ray Serve LLM | Off | Document | 6,310 | 37.9926% | 47.8033% | 16.0383% |
| Ray Serve LLM | Off | Long Generation | 550 | 16.2249% | 18.2488% | 9.9025% |
| Ray Serve LLM | Off | Short Chat | 4,956 | 9.3828% | 10.3936% | 6.1609% |
| Ray Serve LLM | Off | Standard Chat | 2,129 | 21.3150% | 23.3194% | 13.4870% |
| Ray Serve LLM | On | Conversation History | 474 | 19.8272% | 20.1244% | 14.6564% |
| Ray Serve LLM | On | Document | 6,310 | 44.4116% | 48.0846% | 28.3475% |
| Ray Serve LLM | On | Long Generation | 550 | 17.0242% | 18.3235% | 13.0031% |
| Ray Serve LLM | On | Short Chat | 4,956 | 10.0711% | 10.5813% | 7.2067% |
| Ray Serve LLM | On | Standard Chat | 2,129 | 22.3865% | 23.2947% | 15.6075% |
Queue telemetry
| Framework | Prefix caching | Queue scope | Queue wait p50 (s) | Queue wait p95 (s) | Queue wait p99 (s) | Queued requests p50 | Queued requests p95 | Queued requests peak |
|---|
| Dynamo Planner | Off | Dynamo request plane and frontend HTTP queue | 0.000087 | 0.000228 | 0.000250 | 5.00 | 12.00 | 18.00 |
| Dynamo Planner | On | Dynamo request plane and frontend HTTP queue | 0.000089 | 0.000219 | 0.000238 | 4.00 | 8.00 | 16.00 |
| Narwhal | Off | Narwhal admission queue | 0.000363 | 9.305466 | 14.355020 | 0.00 | 0.00 | 4.00 |
| Narwhal | On | Narwhal admission queue | 0.000266 | 9.336533 | 15.216055 | 0.00 | 0.00 | 4.00 |
Definitions
-
Percentages for completed, failed, refused and SLO delivery use all offered requests as the denominator.
-
Natural-completion and output-cap percentages use completed requests as the denominator.
-
Every offer carries a document reference, so every offer has an F1 score. All-offer token F1 averages normalized token-overlap F1 over all offered requests. Failed and refused offers contribute zero.
-
Completed-only token F1 is the mean over completed offers.
-
SLO-delivered token F1 sums F1 from SLO-qualified offers and divides by all offered requests.
-
An SLO-qualified request completed naturally with a nonempty answer and at least two observed tokens. Its TTFT was at or below 2.1 seconds and its TPOT at or below 45 milliseconds.
-
E2E latency rows cover only configurations with completion timing.
-
Queue telemetry rows cover only configurations with sampled server metrics.
-
Failure stage uses observed output-token counts. Terminal signatures come from the raw AIPerf records.
Athrael.net