Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop wrapping single block requests in CARs and graph requests #289

Open
lidel opened this issue Mar 20, 2023 · 1 comment
Open

Stop wrapping single block requests in CARs and graph requests #289

lidel opened this issue Mar 20, 2023 · 1 comment

Comments

@lidel
Copy link

lidel commented Mar 20, 2023

Problem

iiuc L1 always asks Lassie for a CAR:

lassieUrl.searchParams.set("format", "car");

if (isRawFormat) {
await getRequestedBlockFromCar(metricsStream, res, cidObj, blockFilename);

This is very unfortunate when the original request was for a single block adds cost of unnecessary CAR serialization and serialization, and creates surface for amplification bugs which are then worked around with things like #288 (and was the real reason why we did amplification attack against elastic IPFS last week).

Proposed fix

L1 knows if a request was for a single block (inspecting Accept: application/vnd.ipld.raw or ?format=raw), and in such case should ask Lassie the same response type. There should be no CAR.

@lidel lidel changed the title Fetching a single block from Lassie as a CAR is wasteful and error-prone Stop wrapping single block requests in CARs and graph requests Mar 20, 2023
@guanzo
Copy link
Collaborator

guanzo commented Mar 21, 2023

Lassie only returns CARs.

AFAIK there's no way to explicitly request single blocks, like you can with the IPFS Gateway's "raw" endpoint. Rather, you can specify a "shallow" depth which gets translated into a MatchUnixFSPreloadSelector, which may or may not return a single block, depending on if the CID is a directory or not.

#288 (now running in prod) defaults requests to a shallow depth, before it was full depth. That's the minimal request the L1 can make with the current Lassie API. Lassie metrics show that download bandwidth has reduced by 4x for P90, and up to 32x!!! for P95 requests.

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants