AI Lab
Research notes and experiment results from the AVELIN team. Each publication is a page on this site with the full paper attached as a PDF.
Publications
- PDFA Cold Prompt Takes 3.0 Times as Long as a Cached One at 200K to 500K Tokens (2.4 to 3.7)For one operator's coding agents and replies of 300 tokens or fewer: a cold request (under 10% read from cache) takes 3.0 times as long as a warm one at 200K to 500K prompt tokens (95% interval 2.4 to 3.7, 2.7 matched on provider and model). At an assumed read price of a tenth, a miss has 9.8 times the input cost of a hit per prompt token. At 200K or more, 99.2% of requests had a live cache when the same session, provider and model was used under 1 minute earlier, 48.5% at 64 minutes or more.
- PDFSpeculative Decoding on One Production Server: 66% of Draft Tokens Accepted in 29 HoursSpeculative decoding on one production server’s live traffic, described by its operator as internal agentic sessions: in 29 hours it accepted 66.0% of 13.7M draft tokens, between two groups of controlled runs (53.0 to 54.5% and 70.4 to 70.6%). Four of five busy ten-second intervals lay between 56.2 and 75.7%; the mean was flat from one to six or more requests running (at most 13 seen). The geometric formula fed the first-position rate is within 1.1%, two of its four terms exact by construction.
- PDFReading a Long Cold Prompt Starves Generation: About 2% of Normal Speed at an 8,192-Token ChunkWhile an inference server reads a long cold prompt, the requests that are generating slow to a crawl. In a controlled test eight streams kept about 2% of their speed while three prompts of 217K tokens were read at a chunk of 8,192 tokens; in one restarted run a chunk four times smaller gave 3.2 to 4.1 times the generation and 22% slower reading. Over 26 hours of production, 21% of busy ten-second intervals ran at a few percent of normal generation.
- PDFNVFP4 Versus MXFP4 on Synthetic Weights: The Scale's Precision Alone Closes Two Thirds of the Error Gap, the Smaller Block Alone a Quarter at MostMXFP4 and NVFP4 both store 4-bit weights with one shared scale per block. On synthetic Gaussian weights their relative error is 11.50% and 9.51% with the scale from the block maximum, 11.16% and 8.12% with a search. The FP8 scale alone closes two thirds of the gap; the smaller block alone a quarter at most, and almost nothing on Gaussian data. On real matrices of two models (73,920 under the search) mean NVFP4 error stays within 0.04 points of the Gaussian figures.
- PDFCalibration Leakage in a Layer-Wise Quantization Benchmark: With Fewer Calibration Rows Than Input Channels, In-Sample Scoring Understates GPTQ Error 2.6×Scoring a quantization method on the rows it was fitted on understates its error. On 192 matrices of a 27B-parameter language model, encoded to 4-bit NVFP4 by nine encoder settings, GPTQ's output error is 2.28% on its own calibration rows, fewer than the layers' input channels, and 5.94% on held-out rows, a factor of 2.6. Its lead over a plain scale search shrinks from 66% to 13%, and on the widest layers it falls behind a simpler method at the default damping.
- PDFWhere the Time of Long-Context Requests Goes on One Production Server: 62% Generating, 16% Reading the Prompt, 17% WaitingWhere the time of 18,364 requests goes on one production inference server with a context window of a million tokens that serves agentic sessions, from the server's own counters over 31.5 hours. 62% of the summed request time is generating, 16% reading the prompt and 17% waiting in the queue. The first token comes within a second for 71% of requests; the 8% that take more than ten seconds hold at least 61% of all time to the first token.
- PDFLong Prompts of One Operator's Coding Agents: 19% of Requests Carry 200K Tokens or More, Two or More Such Requests Overlap 9% of the TimeA study of 104,088 requests made by one operator's coding agents through one gateway, from token counts alone. The median prompt is 74K tokens; 19.3% of requests carry 200K or more and hold 60% of all prompt tokens. Where cache counters exist (70% of requests), 94.1% of prompt tokens are cache reads. Two or more requests of 200K or more are in flight 8.9% of the time, with a peak of 11.
- PDFA Host-RAM Tier for the KV Cache: Up to 33× Sooner After Eviction, 16% of Cache Misses Served in ProductionAn evicted conversation normally has to be recomputed. With each KV-cache block also kept in host RAM, a conversation of about 200K tokens returned in 1.37 and 0.78 seconds instead of 44.62 and 12.37. Over 26 hours of production on one H200 server, the RAM pool supplied 41 million tokens in 517 restores, 16% of what the GPU cache had missed, for 24 seconds of transfer.
Questions about the AI Lab
What is published in the AI Lab?
Research notes and results of experiments run by the AVELIN team.
Is there a PDF for each publication?
Yes. Each publication has the full paper attached as a PDF, with a download link on the page.
Can I cite or share a publication?
Yes. Link to the page or to the PDF. Each page lists its authors and publication date.
How is the AI Lab different from News?
News covers company and product announcements. The AI Lab holds research write-ups and experiment results.