Inline Images in Text Layouts?

[terhechte]

I was wondering if there was interest in having inline images support. I think it would be reasonably easy for the CoreGraphics backend by adding a NSTextAttachment with the attachment property.

However, I don't know how difficult this would be on the other platforms. I did a brief search for Cairo and found this. However, as with the blurred rectangles maybe it is fine to start just with one (well, two if counting iOS) platform(s) here?

[cmyr]

My sense is that this is out of scope for the time being, as doing it correctly would require us to basically implement our own run-based layout API, which is a lot of work.

I think of the current version of piet as "good enough", in that it can handle the 2D drawing requirements of most simple applications, and once we get into fancier stuff then the capabilities and API of the underlying platforms start diverging rapidly, and there is diminished return on our work. In the long term, our vision for cross-platform 2D graphics is a single unified API that talks directly to the GPU, and avoids intermediate platform API completely; this would give us control over the API we could expose, without worrying about the particulars of the platform.

[raphlinus]

I definitely would like to have a way to do this, but to say it's a complex topic is an understatement. Is it just images, to be drawn when the text layout is drawn, or is it a more general mechanism, for example analogous to replacement spans in Android? Does it allow for animation of the drawn graphic? At some point, you'll also want more sophisticated layout than just inline, for example floats.

As Colin says, one way to address this is to expose a lower level API at the piet level, so the upper layer can draw the runs and the graphics wherever they like. That was originally the idea of skribo, but now I'm not so sure; there are a lot of complexities, and platform text drawing APIs are not designed around to expose this. We will probably be moving away from platform text and toward parley, so it might be worth addressing at that level.

If I had to do something right now, it'd probably be introducing a very simple replacement span concept, where essentially just a box is provided, which would take space in the layout, then the client would be responsible for drawing the contents of the box.

[terhechte]

Thanks for the insightful responses. Just for context, I'm using piet in a new project where (in the long run) I'd need inline images support - if the project bears fruit. Hence my interest in the topics. I don't need it right now though, so I might just as well wait for Piet-gpu. If the need arises I might just fork Piet for the time being and add rudimentary, temporary support to the CoreGraphics backend.

[cmyr]

I think that a coregraphics-specific extension would be the right choice. I also think that the piet API should be quite stable for the foreseeable future, so maintaining a fork shouldn't be that hard.