diff --git a/README.md b/README.md index 4bd39ab..9682ee1 100644 --- a/README.md +++ b/README.md @@ -25,11 +25,3 @@ $ hugo new docs/prologue/TITLE/index.md ``` -### Blogging - -Add a new blog, execute: - -```sh -$ hugo new blog/TITLE/index.md -``` - diff --git a/config/_default/menus.toml b/config/_default/menus.toml index d650c49..f5ae404 100644 --- a/config/_default/menus.toml +++ b/config/_default/menus.toml @@ -18,11 +18,6 @@ url = "/reference" weight = 40 -[[main]] - name = "Blog" - url = "/blog/" - weight = 50 - [[main]] name = "GitHub" url = "https://github.com/benbjohnson/litestream" diff --git a/content/blog/_index.md b/content/blog/_index.md deleted file mode 100644 index fe3d8ce..0000000 --- a/content/blog/_index.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -title: "The Litestream Blog" -date: 2021-02-01T00:00:00Z -draft: false -images: [] ---- diff --git a/content/blog/v0.3.3-released/index.md b/content/blog/v0.3.3-released/index.md deleted file mode 100644 index 7148667..0000000 --- a/content/blog/v0.3.3-released/index.md +++ /dev/null @@ -1,205 +0,0 @@ ---- -title: "Litestream v0.3.3 Released" -description: "The newest version of Litestream brings support for Windows and support for S3-compatible storage such as MinIO. It also includes a dozen quality-of-life fixes and expands documentation." -date: 2021-03-16T00:00:00Z -draft: false -weight: 50 -contributors: ["Ben Johnson"] ---- - -The newest version of Litestream brings support for Windows and support for -S3-compatible storage such as MinIO. It also includes over a dozen -quality-of-life fixes and expands documentation. - -Beyond code changes, we've opened up a Slack channel & started office hours to -help folks get up and running. Let's run through some of these changes. - - -## Slack & office hours - -While Litestream has an active [GitHub Discussions board][discussion] and -[Issues section][issues], sometimes you want to have synchronous communication -to quickly figure out an issue or just to get up and running. - -We now have our own [Litestream Slack][slack] with over two dozen Litestreamers -(or Litestreamites or whatever name you might call us) chatting about SQLite and -how to best configure, run, and backup our applications. It's been a great way -to quickly discuss new features, hear about use cases, and talk about -documentation changes. - -But sometimes text communication can be limiting so I've introduced weekly -office hours for Litestream every Friday. [Sign up on Calendly][calendly] to -book a 20 minute spot to discuss anything SQLite or Litestream related. I've -already had a great discussion with [Simon Willison][simonw] about his -[Datasette][datasette] project where we brainstormed ways to improve performance -and to scale out. - -[discussion]: https://github.com/benbjohnson/litestream/discussions -[issues]: https://github.com/benbjohnson/litestream/issues -[slack]: https://join.slack.com/t/litestream/shared_invite/zt-n0j4s3ci-lx1JziR3bV6L2NMF723H3Q -[calendly]: https://calendly.com/benbjohnson/litestream -[simonw]: https://twitter.com/simonw -[datasette]: https://datasette.io/ - - -## Windows support - -Litestream now ships with prebuilt binaries for Windows so you can run it as a -standalone application or as a background Windows Service. It integrates in -with the Windows Event Log so you can it can collect and manage log events. - -You can find a guide for running [Litestream as a Windows Service][windows] in -the documentation section of the web site. We'd love to hear feedback on the -setup and if anything could be improved. - -[windows]: http://localhost:1313/guides/windows/ - - -## S3-compatible storage - -AWS S3's API has become the de facto API for object storage and most platforms -support it. The previous version of Litestream was built to use this API but it -needed a few tweaks to support all S3-compatible storage solutions. It also adds -some built-in support for several popular services so they can easily be used as -a replica URL in Litestream. - -For this release, Litestream has been tested on [MinIO][minio], [Backblaze -B2][b2], [DigitalOcean Spaces][do], & [Linode Object Storage][linode]. - -So why would you choose a different object storage backend besides AWS S3? It -depends on your use case. - -MinIO works great if you need to manage your own object storage solution. It's -easy to get started—so easy that we've updated our [_Getting Started_][gs] -section to use it. - -Backblaze B2 is a great solution if you want to reduce your AWS S3 spend. It's -pricing is significantly cheaper than the major cloud providers. Litestream is -already quite cheap to run on AWS but you may already have a Backblaze account -for other storage. - -DigitalOcean & Linode object storage have been integrated because many -developers already run on these platforms since they provide low-cost VPS -servers. Adding in their object storage platforms makes it easy to run -everything in one place. - -You can find walkthroughs for each of these platforms in our [Guides][guides] -section. - -[minio]: https://min.io/ -[b2]: https://www.backblaze.com/b2 -[do]: https://www.digitalocean.com/products/spaces/ -[linode]: https://www.linode.com/products/object-storage/ -[gs]: /getting-started/ -[guides]: /guides/ - - -## Snapshot interval - -Litestream works by backing up a point-in-time snapshot of your database and -then incrementally backing up new write-ahead log (WAL) frames. When you need -to restore your database, Litestream pulls down that snapshot and then replays -every WAL frame that has been written since. You can set a retention period so -that your database will re-snapshot periodically and remove old WAL frames. - -That normally works well but sometimes you may wish to snapshot more often but -still retain changes for a longer period. This helps improve recovery time as -the database doesn't need to replay as many WAL frames. - -The newest version of Litestream now includes a separate ["snapshot interval" -configuration setting][snapshot]: - -```yaml -dbs: - - path: /path/to/db - replicas: - - url: s3://mybkt.litestream.io/db - retention: 24h - snapshot-interval: 4h -``` - -In this example configuration file, we are keeping our backups for 24 hours but -we'll re-snapshot the database every 4 hours. This means that we'll have 6 full -snapshots of our database in S3 at any given time. When we restore, we'll only -need to restore up to four hours of WAL frames instead of a full day's worth. - -[snapshot]: https://github.com/benbjohnson/litestream/pull/84 - - -## Quality-of-life improvements - -The primary goal of Litestream is to keep your SQLite data safely backed up -without you having to think about it too much. Set it and forget it. To this -end, quality-of-life improvements and documentation are top priorities because -they make running Litestream even easier. - -Some of these improvements include [making smarter defaults with the -configuration][120] and [preventing misuse of the config & command line flags][94], -You can read the full changelog of improvements on the [release page][release]. - -[94]: https://github.com/benbjohnson/litestream/pull/94 -[120]: https://github.com/benbjohnson/litestream/pull/120 -[release]: https://github.com/benbjohnson/litestream/releases/tag/v0.3.3 - - -## Bug fixes - -The previous version of Litestream has generally been stable but there were a -few bugs that folks encountered that have been fixed. - -### Failure to retry on init error - -There was a bug where Litestream would would not attempt to [reinitialize -the connection with a database if it failed on startup][82]. Database failures -can happen for many reasons and many of them are temporary so Litestream retries -all actions periodically. Litestream will now correctly retry even if an error -occurs on startup. - -[82]: https://github.com/benbjohnson/litestream/pull/82 - - -### Fix locks on operating systems without OFD locks - -[Open file descriptor (OFD) locks][ofd] are locks that are handled per file -descriptor. If you lock a file descriptor then unlocking the same file on a -different descriptor won't affect the first one. That makes sense. - -Unfortunately, that's not how all operating systems work. Some operating systems -lock per-process so you can mess up your locks if you have multiple file -descriptors for the same file. - -This is a problem for Litestream as it requires a lock on the file to prevent -SQLite from deleting the WAL file. The lock handling has been [fixed for non-OFD -lock operating systems][93] so they are now safe to use. - -[ofd]: https://www.gnu.org/software/libc/manual/html_node/Open-File-Description-Locks.html -[93]: https://github.com/benbjohnson/litestream/pull/93 - - -## Conclusion - -The latest version of Litestream includes huge improvements for usability and -helping folks run it on more systems. But the work isn't done yet! - -Upcoming work on the next version of Litestream (v0.3.4) will include better -support for ARM architectures and documentation on running Litestream in Docker -& Kubernetes. You can find a full list of upcoming work on the [v0.3.4 -milestone][v0.3.4] - -The next major work will be on [live read replication][replica] for Litestream -v0.4.0. This will allow users to scale out their single-node SQLite database to -multiple servers. - -_Yes, you will be able to run a cluster of SQLite servers!_ - -Sounds ridiculous, sure, but there are actually some really great use cases. -First, it lets you cheaply scale up a service to multiple servers to improve -your query performance. Second, it opens up some exciting possibilities -for moving read queries to the edge to get extremely low latency requests. - -If you're interested in hearing more, please [join our Slack][slack], [book an -office hours][calendly], or [follow Litestream on Twitter][twitter] for updates. - -[v0.3.4]: https://github.com/benbjohnson/litestream/milestone/5 -[replica]: https://github.com/benbjohnson/litestream/issues/8 -[twitter]: https://twitter.com/litestreamio diff --git a/content/blog/v0.3.4/index.md b/content/blog/v0.3.4/index.md deleted file mode 100644 index 26a64b3..0000000 --- a/content/blog/v0.3.4/index.md +++ /dev/null @@ -1,311 +0,0 @@ ---- -title: "Litestream v0.3.4 Released" -description: "The newest version of Litestream updates its license, adds Docker & ARM support, and improves recovery performance." -date: 2021-04-28T00:00:00Z -draft: false -weight: 50 -contributors: ["Ben Johnson"] ---- - -Over the last month, we've been focused on stability & quality-of-life changes -to Litestream. We now have an [official Litestream Docker -image][litestream-docker] so you can run your backups as a sidecar to your -existing Docker image. We've also provided an [example Dockerfile][Dockerfile] -for those wanting to bundle their application with Litestream into a single -Docker container using [s6][s6-overlay]. - -We've added support for ARMv6, ARMv7, & ARM64 which means you can take advantage -of Litestream on a tiny [Raspberry Pi Zero][rpi] or on a beefy [64-core Graviton -EC2 machine][graviton]. - -Our license has changed as well! We've move from a restrictive, copyleft GPLv3 -license to a permissive Apache 2 license. - -Finally, we've significantly improved the recovery performance and fixed up a -handful of bugs. Thanks to everyone who reported bugs, gave feedback, and helped -with testing! - -[Dockerfile]: https://github.com/benbjohnson/litestream-s6-example/blob/main/Dockerfile -[s6-overlay]: https://github.com/just-containers/s6-overlay -[rpi]: https://www.raspberrypi.org/products/raspberry-pi-zero/ -[graviton]: https://aws.amazon.com/ec2/graviton/ - - -## Docker support - -The most frequent question asked about Litestream since the project started has -been, _"how do I run this on Docker?"_ Whether you like Docker or not, the fact -is that much of the tech world has moved to containerized deployments. - -In traditional applications, developers deploy their stateless, single-process -application within a Docker container and it connects to a stateful database. -However, with Litestream, not only is the container stateful but Litestream also -runs as a separate process. - -There are a couple ways to Dockerize your application depending on your needs. -Let's take a look at some common use cases. - - -### Shared volume, sidecar replication - -The first use case is to keep your application as a single-process container -and mount a persistent volume for your database. You can then use the [official -Litestream Docker image][litestream-docker] ([#156][]) to mount the -same volume and replicate from there. - -You can find documentation for running a shared volume in the [_Running as a -sidecar_][sidecar] section of our [Docker guide][docker-guide]. - -[litestream-docker]: https://hub.docker.com/r/litestream/litestream/ -[sidecar]: /guides/docker/#running-as-a-sidecar -[docker-guide]: /guides/docker/ -[#156]: https://github.com/benbjohnson/litestream/pull/156 - - -### Single container, multiple processes - -Not all deployments support a sidecar container. For example, [fly.io][] -provides an excellent, simple platform for deploying applications but the caveat -is that those applications need to be in a single container. - -Fortunately, there's a great solution called [s6-overlay][] for running your -application as well as Litestream in a single container. It uses a simple init -system called [s6][] that forwards signals, restarts background processes, and -allows you to teardown the container when your main application finishes. - -You can run this setup with a mounted volume to persist your database between -deployments or you can run this ephemerally so that your database is restored -from S3 on each deploy. Personally, I recommend using a persistent disk as it -will minimize downtime during deployments. However, if you have a small, -low-write database then an ephemeral disk can work just fine. You can find -an example of this ephemeral method on Michael Lynch's [logpaste][] project. - -You can find an example application & walkthrough on the -[litestream-s6-example][] repository. - -[fly.io]: https://fly.io/ -[s6]: http://skarnet.org/software/s6/overview.html -[litestream-s6-example]: https://github.com/benbjohnson/litestream-s6-example -[logpaste]: https://github.com/mtlynch/logpaste - -### Making Docker sing - -Building a Dockerfile for Litestream was pretty straightforward, however, making -it something developers want to use is an entirely different story. We've added -a bunch of small improvements that make deploying much simpler. - -First, we've added a `-if-replica-exists` flag to the `restore` command -([#131][]). This will cause Litestream to exit successfully if there are no -replicas available to restore from. This simple change makes it easy to create -an initialization script that will restore if a database replica is available -but otherwise will let the application create a new database on the first run. - -Next, we fixed up the shutdown process for Litestream so that it will flush all -outstanding writes ([#132][]) and it will correctly catch `SIGTERM` ([#133][]). -These two fixes allow users to run an empheral disk to help ensure recent writes -get backed up. - -In the Docker world, environment variables work better than configuration files, -however, Litestream needs configuration files for complex configuration options. -To support both of these options, we've added environment variable expansion in -configuration files ([#157][]). This means you can bundle a config into your -container and define your own environment variables that will be evaluated -before your configuration file is read. - -Finally, we've shrunk down the binary sizes on the release builds ([#159][]) and -have included static builds in our releases ([#130][]). Docker container sizes -can easily explode into the hundreds of megabytes or even gigabytes. The Linux -amd64 binary for Litestream is now 17.4MB. - -[#130]: https://github.com/benbjohnson/litestream/pull/130 -[#131]: https://github.com/benbjohnson/litestream/pull/131 -[#132]: https://github.com/benbjohnson/litestream/pull/132 -[#133]: https://github.com/benbjohnson/litestream/pull/133 -[#157]: https://github.com/benbjohnson/litestream/pull/157 -[#159]: https://github.com/benbjohnson/litestream/pull/159 - - -## ARM support - -SQLite is sometimes called an "edge database" in that it can be run on small ARM -devices far away from data centers. Not only do these devices have limited -resources but can also have intermittent internet connections. - -We've added release builds for 32-bit ARM (`arm6`, `arm7`) and 64-bit ARM -(`arm64`) ([#148][], [#151][]). The 32-bit ARM processors are typically found in -smaller or older devices like the [Raspberry Pi Zero][rpi] while the 64-bit -processors are gaining popularity in data centers such as AWS' new [Graviton -series][graviton]. - -Apple recently released their Apple Silicon M1 chip and we do not have a release -build for it yet but we are planning to add it in the future. If you are -interested in using Litestream with the new Macs, please add a comment or add a -reaction on [issue #175][#175] and that'll help us prioritize it. - -[#148]: https://github.com/benbjohnson/litestream/pull/148 -[#151]: https://github.com/benbjohnson/litestream/pull/151 -[#175]: https://github.com/benbjohnson/litestream/pull/175 - - -## Open & permissive licensing - -When Litestream first started, it used [GPLv3][]. The GPL is known for what's -called a "copyleft" provision which means that anyone who changes the code or -creates a derivative work needs to release their code as open source as well. -I've always used very permissive licenses (e.g. [MIT][]) in other projects but I -felt that I could try using GPL for Litestream because it's a standalone -library. - -However, I found the license to be overly restrictive. While Litestream started -as a separate standalone tool, I would like to release it as an embeddable -library in the future. A GPL license would require anyone who includes -Litestream in their application to also release their code as open source which -seemed unreasonable. Even an LGPL license would be restrictive as Go uses -static compilation. - -I've had many people theorize why I open sourced Litestream—some think I want -to make it into a business, others think I want acclaim. The reason I open -sourced Litestream is actually quite simple—I think the best software is -software that gets used. Open source is a fantastic means of distribution and -the more people that use Litestream, give feedback, & share ideas, the better -it will be. - -The GPL is antithetical to my goal. It restricts usage rather than encourages -it. Because of that, I've changed Litestream to an [Apache 2][APLv2] license -([#168][]). After researching and asking around, I found it to be the best -choice for a permissive license. While the MIT & BSD licenses are wonderful in -their simplicity, the Apache 2 license seems to be more friendly to lawyers who, -at the end of the day, are really who licenses are for. - -[GPLv3]: https://www.gnu.org/licenses/gpl-3.0.en.html -[MIT]: https://opensource.org/licenses/MIT -[APLv2]: https://www.apache.org/licenses/LICENSE-2.0 -[#168]: https://github.com/benbjohnson/litestream/pull/168 - - -## Improved restore performance - -Hopefully you don't ever need to restore your database because of a catastrophic -event but, when you do, you want it to be fast. Litestream's recovery process -involves downloading a snapshot of your database and then replaying all -subsequent WAL files into that database. - -Previously, Litestream would download a WAL file and then apply it, download -another file and apply it, etc. The download time would typically dominate the -restore time because applying the WAL file may only take a couple milliseconds. -The constant switching also meant that there wasn't always a file being -downloaded so the network device would not be saturated. - -The latest version of Litestream now downloads WAL files in parallel and queues -them up on disk to be processed ([#167][]). By default, it will download 8 files -in parallel. This change has improved recovery time by 3-4x. You can increase -the number of parallel downloads by setting the `-parallelism` flag: - -```sh -$ litestream restore -parallelism 16 /path/to/db -``` - -If you need faster recovery, you should also consider setting the -`snapshot-interval` for your replica to a lower duration (e.g. `1h`). - -There was also an poor usability when recovering by index. Previously, the -`-index` flag in the `restore` command would accept a decimal number, however, -Litestream reports the index as a hexadecimal number everywhere else. We've -fixed it so that `-index` now accepts the hex-formatted number ([#165][]): - -```sh -$ litestream restore -generation xxxxxxxxxxxxxxxx -index 000002bc /path/to/db -``` - -We've also improved logging for restores to help identify bottlenecks ([#162][], -[#164][]), removed the `-dry-run` flag as it complicated the code ([#163][]), -and fixed a bug around snapshot selection when recovering by index ([#166][]). - -[#162]: https://github.com/benbjohnson/litestream/pull/162 -[#163]: https://github.com/benbjohnson/litestream/pull/163 -[#164]: https://github.com/benbjohnson/litestream/pull/164 -[#165]: https://github.com/benbjohnson/litestream/pull/165 -[#166]: https://github.com/benbjohnson/litestream/pull/166 -[#167]: https://github.com/benbjohnson/litestream/pull/167 - - -## Reduced lock contention - -Litestream is able to run as a separate process because SQLite is built as a -multi-process database. However, applications typically need to set a timeout on -their connection called `busy_timeout` to avoid returning an error if another -process obtains a brief write lock at the same time they want a write lock. You -can read more about it in our [Tips & Caveats](/tips) section on our site. - -Originally, Litestream would frequently obtain a very brief write lock to ensure -the WAL was at the end of a transaction before copying it to Litestream's shadow -WAL. However, we later updated Litestream to validate transaction boundaries -itself so this lock was no longer needed. We updated Litestream to remove this -lock ([#104][]) but found some timing issues when obtaining snapshots so we had -to remove it ([#108][]). - -We've done more testing and have been able to successfully remove the lock again -by adding some additional read locks around snapshots ([#170][]). While there -is still a write lock during checkpointing, write locks will be significantly -less frequent. - -[#104]: https://github.com/benbjohnson/litestream/pull/104 -[#108]: https://github.com/benbjohnson/litestream/pull/108 -[#170]: https://github.com/benbjohnson/litestream/pull/170 - - -## Other bug fixes - -First of all, thanks to everyone who reports bugs to our [GitHub repo][gh]. It -makes a world of difference and we prioritize those to be fixed first. In this -relase, we've fixed the following bugs & usability issues: - -- Added `skip-verify` to replicas to allow use with MinIO self-signed certificates ([#141][]) -- Removed confusing reference to "wal" in error message when invalid databases are opened ([#154][]) -- Omitted load extensions from static builds to avoid warning messages ([#158][]) -- Fix flakey WAL filesystem timestamp test ([#161][]) - -We've also started allowing for setting `LITESTREAM_ACCESS_KEY_ID` & -`LITESTREAM_SECRET_ACCESS_KEY` environment variables instead of the original -`AWS` prefixed versions ([#169][]). The AWS-prefixed versions still work but -some users where confused to use those for non-AWS S3-compatible services: - -```sh -export LITESTREAM_ACCESS_KEY_ID=AKIAxxxxxxxxxxxxxxxx -export LITESTREAM_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -``` - -[gh]: https://github.com/benbjohnson/litestream -[#141]: https://github.com/benbjohnson/litestream/pull/141 -[#154]: https://github.com/benbjohnson/litestream/pull/154 -[#158]: https://github.com/benbjohnson/litestream/pull/158 -[#161]: https://github.com/benbjohnson/litestream/pull/161 -[#169]: https://github.com/benbjohnson/litestream/pull/169 - - - -## Conclusion - -We continue to focus on stability & usability with Litestream v0.3.4. We've -expanded our deployment options to include Docker containers & ARM processors, -we've opened up our code by switching to a permissive Apache 2 license, and -we've significantly improved our time to restore. - -Upcoming work on the next version of Litestream ([v0.3.5][]) will include a -refactor of the replica system to make it easier to add new replica types -([#135][]). This will enable better support for major cloud vendor storage such -as Azure Blob Storage ([#134][]) & Google Cloud Storage ([#69][]) as well as -older protocols such as SFTP ([#140][]). - -If you're interested in hearing more, please [join our Slack][slack], [book an -office hours][calendly], or [follow Litestream on Twitter][twitter] for updates. - -[#135]: https://github.com/benbjohnson/litestream/pull/135 -[#134]: https://github.com/benbjohnson/litestream/pull/134 -[#69]: https://github.com/benbjohnson/litestream/pull/69 -[#140]: https://github.com/benbjohnson/litestream/pull/140 -[v0.3.5]: https://github.com/benbjohnson/litestream/milestone/6 -[replica]: https://github.com/benbjohnson/litestream/issues/8 -[slack]: https://join.slack.com/t/litestream/shared_invite/zt-n0j4s3ci-lx1JziR3bV6L2NMF723H3Q -[calendly]: https://calendly.com/benbjohnson/litestream -[twitter]: https://twitter.com/litestreamio diff --git a/content/blog/v0.3.5/index.md b/content/blog/v0.3.5/index.md deleted file mode 100644 index 5a55fec..0000000 --- a/content/blog/v0.3.5/index.md +++ /dev/null @@ -1,281 +0,0 @@ ---- -title: "Litestream v0.3.5 Released" -description: "The newest version of Litestream adds a slew of new replication destinations, improves Kubernetes integration, & standardizes the S3 sync interval." -date: 2021-06-03T00:00:00Z -draft: false -weight: 50 -contributors: ["Ben Johnson"] ---- - -The main focus of this latest release has been on improving the replica code so -we can easily add new replication destinations. Additionally, we've improved -Docker integration, made quality-of-life improvements for running -Litestream in Kubernetes, adjusted the default S3 sync interval, updated -some metrics, and made a bunch of smaller bug fixes. - - -## Improved replica code - -Originally, Litestream only had `s3` and `file` replica types. These -had a lot of similar code but there were enough small differences in how they -behaved that we didn't try to abstract the code too much. For example, `file` -replicas don't need to worry about batching WAL segments because writes are -free. In contrast, S3 charges $0.000005 per PUT request which can add up with -frequent uploads. - -After working with both implementations and planning out future replica -implementations, it became clear how to separate them. Litestream now has a -`Replica` type for managing WAL position, snapshot frequency, & retention while -the new `ReplicaClient` type provides a "dumb" client interface to the external -replica. This client is "dumb" in the sense that it is entirely stateless so it -makes it easy to add new client types. - - -### Azure Blob Storage - -[Azure][] is one of the largest cloud providers after AWS and, sadly, does not -support the S3 API in their blob storage service. Users could add a translation -layer to convert S3 requests to Azure's API but that's a poor experience. - -We've added native support for [Azure Blob Storage][] in the latest release. It -uses the [official Azure SDK](https://github.com/Azure/azure-sdk-for-go) and you -can use it by specifying the `abs` replica type. - -If you're using replica URLs, you can specify it like this: - -``` -abs://STORAGEACCOUNT@CONTAINERNAME/PATH -``` - -However, if you use the Litestream [config file][config] then you can break it out like -this: - -```yml -dbs: - - path: /path/to/local/db - replicas: - - type: abs - account-name: STORAGEACCOUNT - account-key: ACCOUNTKEY - bucket: CONTAINERNAME - path: PATH -``` - -You'll need to set the `LITESTREAM_AZURE_ACCOUNT_KEY` environment variable if -you are using the URL format. You can find more details on our [_Replicating to -Azure Blob Storage guide_](/guides/azure). - -[Azure]: https://azure.microsoft.com/en-us/ -[Azure Blob Storage]: https://azure.microsoft.com/en-us/services/storage/blobs/ -[config]: /reference/config - -### Google Cloud Storage - -[Google Cloud Storage] was originally supported in Litestream via its -S3-compatible API, however, it turns out that it's not all that compatible. -We've integrated the native [Google Cloud SDK][] to provide a replica client -for GCS which you can specify using the `gcs` replica type. - -The replica URL format looks like: - -``` -gcs://BUCKET/PATH -``` - -While the configuration file usage looks like: - -```yml -dbs: - - path: /path/to/local/db - replicas: - - type: gcs - bucket: BUCKET - path: PATH -``` - -You'll need to make sure the path to your service account key JSON file is set -to the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. You can find more -details on our [_Replicating to Google Cloud Storage guide_](/guides/gcs). - -[Google Cloud Storage]: https://cloud.google.com/storage -[Google Cloud SDK]: https://cloud.google.com/go/storage - - -### SFTP - -If there's one thing we love it's old school technology. Tried & true. SQLite is -a perfect example! - -[Mia Bennett](https://github.com/chillfox) posted on the Litestream [discussion -board][] to suggest adding SFTP support to be able to replicate to [rsync.net][] -and we jumped on it. Huge shout out to [John Kozubik][] from rsync.net for -providing a test account! - -SFTP is an FTP-like protocol that runs over SSH so it's secure and pretty easy -to setup. It's typically secured via password or SSH key file. You can use both -with Litestream. - -You can specify an SFTP replica destination via replica URL: - -``` -sftp://USER:PASSWORD@HOST:PORT/PATH -``` - -Or you can use the configuration file format: - -```yml -dbs: - - path: /path/to/local/db - replicas: - - type: sftp - host: HOST:PORT - user: USER - password: PASSWORD - path: PATH -``` - -If you are using an SSH key file, you can specify its path in the config only: - -```yml -dbs: - - path: /path/to/local/db - replicas: - - url: sftp://USER@HOST:PORT/PATH - key-path: /path/to/id_rsa -``` - -[rsync.net]: https://www.rsync.net/ -[discussion board]: https://github.com/benbjohnson/litestream/discussions -[John Kozubik]: https://john.kozubik.com - - -## Improved Docker integration - -The previous version of Litestream added quality-of-life improvements to -integrate with Docker by using [s6][] for process supervision. This worked -well but it added an additional layer of complexity for most applications -that only need to run a single application process. - -We've added subprocess execution into Litestream so you can run a single -process as a child of Litestream. This process will be monitored and Litestream -will automatically shutdown when the process exits. - -You can run a child process by providing your command and args with the `-exec` -flag: - -```sh -litestream replicate -exec "myapp -myflag myarg" -``` - -or you can specify it in your config file: - -```sh -dbs: - - path: /path/to/db - exec: myapp -myflag myarg -``` - -You can find a full, working example of a Dockerized application running with -subprocess execution at the [Litestream/Docker example repo][litestream-docker-example]. - -[s6]: https://skarnet.org/software/s6/ -[litestream-docker-example]: https://github.com/benbjohnson/litestream-docker-example - - -## Kubernetes deployments - -Since the early days of Litestream, folks have been asking how to run it as a -sidecar in Kubernetes. It seemed like an odd request since Litestream is meant -to simplify application deployments and [Kubernetes][] is anything by simple. -Litestream also constrains applications to a single node so it complicates -Kubernetes deployments where most pods are stateless. - -However, it turns out there are some great use cases for running Litestream in -Kubernetes. Most applications have peripheral services don't require zero -downtime or extremely high traffic. Worker and monitoring services are good -examples. Being able to run against a local SQLite database greatly simplifies -the deployment & development process. - -[Sam Weston](https://twitter.com/cablespaghetti) did some great work to get -Litestream working in a Kubernetes StatefulSet with a single replica. You can -find details about configuring it and setting up autorecovery on the [_Running -in a Kubernetes Cluster guide_](/guides/kubernetes/). - -[Kubernetes]: https://kubernetes.io/ - - -## S3 sync interval changes - -One of the goals of Litestream was to let you run your application on a single -server and pay almost nothing to replicate it. The goal was to target $1 per -month. S3 charges $0.000005 per upload so if Litestream uploads every 10 seconds -(or 260,000 times per month) then it would cost you about $1.30 per month. So we -picked a default sync interval of 10 seconds. - -However, replicating every 10 seconds is not a great experience when folks were -going through the [Getting Started](/getting-started) tutorial so we changed -the the default to be 1 second if you were running via the command line args. - -This is obviously confusing. - -After some usage by the community, it became obvious that the original 10 second -sync interval was too conservative. Most applications don't have a continuous -stream of writes 24/7. For example, [Michael Lynch][] wrote about his experience -in a [blog post][] where his costs were only $0.03/month. - -So we've standardized the sync interval for S3 to replicate every 1 second -regardless of if you're using the config or command line args. If you do have a -heavy, constant write load then you may want to adjust your replica's -`sync-interval` property to a higher interval. - -[Michael Lynch]: https://twitter.com/deliberatecoder -[blog post]: https://mtlynch.io/litestream/ - - -## Replica metrics - -The original S3 replica had two Prometheus metrics that it published to track -the number of operations and the number of bytes used by those operations: - -``` -litestream_s3_operation_total{"operation"} -litestream_s3_operation_bytes{"operation"} -``` - -However, we have a lot more replica types now and it didn't make sense to have -separate metrics for each one. These have now been consolidated into generic -metrics with a `"replica_type"` label: - -``` -litestream_replica_operation_total{"replica_type","operation"} -litestream_replica_operation_bytes{"replica_type","operation"} -``` - -Note that each replica type reports a little differently because of how the -native library implements LIST, GET, and PUT commands. - - -## Miscellaneous fixes - -We've fixed a few minor bugs in the latest version: - -- Support older versions of Linux ([#199](https://github.com/benbjohnson/litestream/pull/199)) -- Add support for [Filebase](https://filebase.com/) replica URLs ([#210](https://github.com/benbjohnson/litestream/pull/210)) - - -## Conclusion - -The latest release of Litestream greatly expands the number of replica -destinations and provides some quality of life improvements for Kubernetes -users. We're excited for the next release (v0.4.0) where we will be focused on -[live read replicas](https://github.com/benbjohnson/litestream/issues/8) to allow -users to run multiple nodes and distribute query load between them. It creates -some fantastic opportunities for building large read-heavy applications or for -reducing client latency by creating a globally-distributed SQLite database. - -If you're interested in hearing more, please [join our Slack][slack], [book an -office hours][calendly], or [follow Litestream on Twitter][twitter] for updates. - -[slack]: https://join.slack.com/t/litestream/shared_invite/zt-n0j4s3ci-lx1JziR3bV6L2NMF723H3Q -[calendly]: https://calendly.com/benbjohnson/litestream -[twitter]: https://twitter.com/litestreamio