Athrael.netAthrael.net
RSS FeedA blog about everything, everywhere, all at once. 'Pun intended'
Recent Posts
Evaluating Narwhal: Adaptive Prefill/Decode Allocation Under Rapidly Changing Workloads
Updated:On Kimi-K3 with six TP8 engines, Narwhal delivered 76.18% of chat/document requests within the latency SLO, against 56.76% for Dynamo Planner and 51.66% for Ray Serve LLM. The main trade-off was an 11-second p95 TTFT tail on mixed traffic.
The Price of Anarchy in Disaggregated Inference
I split NVIDIA Dynamo's prefill and decode into three competing games and measured the Price of Anarchy on a 3-node B200 cluster. While the GPUs had headroom, no router tuning moved the needle; the moment they saturated, one parameter was the gap between a 1-second tail and a 28-second one. So I built a 270-line controller that watches for that moment and flips the switch, without touching Dynamo's core.
Mine the Way Your Model Scores: MaxSim Hard-Negative Mining for a Late-Interaction Student
Updated:The standard way to mine hard negatives for a late-interaction model uses a single-vector cosine teacher, even though the model itself scores with multi-vector MaxSim. So I rebuilt my miner to score the way my model does. Matched mining clearly beat training with no mined negatives, while the cosine approach was barely doing anything at all.
Diminishing Returns and the Art of Knowing When to Stop
I trained three generations of ColQwen3.5, each with more sophisticated optimization than the last. The most optimized version barely beat the previous one on the primary benchmark (+0.0011 nDCG@5). Individual tasks reshuffled substantially, with per-task swings an order of magnitude larger than the aggregate gain.
Closing the AI Value Gap: Insights from Research
Enterprise AI adoption has reached 88%, yet only 5% of pilots deliver measurable impact. Research from MIT, BCG, and RAND reveals what separates successful implementations from the rest. It's not the technology.