Using Delta Updates in the Astro Content Layer to Reduce API Traffic

Delta Updates

Delta updates is a term originally derived from traditional software that had to be downloaded. Typically, when you wanted to update software to a newer version, you had to download the whole software package again. Delta updates solved this by only downloading the files necessary to upgrade to the newer version, thus saving time and bandwidth.

In the context of a headless CMS, where code and content are decoupled, content is fetched by your code through an API layer provided by your CMS. During development, where rapid changes happen in both content and code, the whole of your content is often re-fetched multiple times, even though only a part of it may have been updated. For example, if you have a blog on Storyblok, and each blog post is a story, every time your code is rebuilt, all the stories will be re-fetched.

We can apply the delta updates paradigm to optimize this. If your content delivery APIs support delta updates, you can use them to ensure that you only load data and content that has changed since the last build or deployment.

Storyblok Content Delivery APIs support delta updates and work well with the Astro Content Layer.

Astro Content Layer

Introduced in Astro v5, the Content Layer is a feature of Astro that facilitates the fetching of external content needed to build a static site. For example, in a blog built with Astro using Storyblok as a CMS, blog posts would be a content collection. In a trivial setup, during development, you would be fetching the entire list of stories (blog posts) every time you rebuild your blog. Since the Content Layer fetches all data once at build time and then caches it, this step is optimized. The Content Layer also allows us to update the content collection selectively, which is where delta updates come in. Let us learn how to do this in Astro.

Setting up

To make the most efficient use of this tutorial, you should have an introductory knowledge of Astro. We will build a blog using full static rendering and Storyblok as a CMS. Use this tutorial on the Astro blog to set up a base for our tutorial. Remember to skip the "On Demand Rendering” step since we will use full static site generation.

Let us now have a look at how we fetch content from Storyblok in the current setup.

In [...slug].astro , which is used to generate pages at build-time dynamically, we are currently fetching all stories from the Storyblok API:

const { data } = await sbApi.get('cdn/stories', {
  content_type: 'blogPost',
  version: import.meta.env.DEV ? 'draft' : 'published',
})

Then, for each individual story, we fetch the entire content of the story as well:

const sbApi = useStoryblokApi()
const { slug } = Astro.params
const { data } = await sbApi.get(`cdn/stories/blog-posts/${slug}`, {
  version: import.meta.env.DEV ? 'draft' : 'published',
})

This means that every time I make a change and refresh the list of blog posts, the Storyblok API is called, and ALL stories are fetched again. We will now augment our code by adding the Astro Content Layer to optimize this.

Adding the Content Layer

The Astro Content Layer is very extensible and can load data from a variety of sources. To do so, you need to define a loader for each content collection. We can define our own loader to load data from the Storyblok API, but luckily, Storyblok has a first-party loader that you can use. It’s included with the @storyblok/astro package. Let’s install the package:

npm i @storyblok/astro@alpha

Once this is done, we will define the config for the Content Layer. Create a new file under src/content/config.js and add the following code:

import { storyblokLoader } from '@storyblok/astro'
import { defineCollection } from 'Astro:content'

const storyblokCollection = defineCollection({
  loader: storyblokLoader({
    accessToken: import.meta.env.STORYBLOK_TOKEN,
    version: process.env.NODE_ENV === 'development' ? 'draft' : 'published',
  }),
})
export const collections = {
  storyblok: storyblokCollection,
}

The above code imports the official Storyblok loader, initializes it with the required config and then defines it as a loader for the collection called storyblok. Save the file, and rerun astro dev.

When you start the astro development server, Astro populates the Content Layer with the data from the Storyblok loader. In our case, it loads and stores all the stories in the content collection. Let us modify our code to load stories from this content collection instead of calling the API again.

Replace the code in [...slug].astro with the following:

---
import StoryblokComponent from "@storyblok/astro/StoryblokComponent.astro";
import { getCollection } from "astro:content";
import { type ISbStoryData } from "@storyblok/astro";

export async function getStaticPaths() {
  const stories = await getCollection("storyblok");

  return stories.map(({ data }: { data: ISbStoryData }) => {
    return {
      // This assumes that the blog posts path is /blog-posts/ and that is where this file is
      params: { slug: data.slug },
      props: { story: data },
    };
  });
}

interface Props {
  story: ISbStoryData;
}

const { story } = Astro.props;
---

<html lang="en">
  <head>
    <title>Storyblok & Astro</title>
  </head>
  <body>
    <StoryblokComponent blok={story.content} />
  </body>
</html>

Replace the code in BlogPostList.js with this:

---
import { storyblokEditable , type ISbStoryData} from "@storyblok/astro";
import { getCollection } from "Astro:content";

const stories = await getCollection("storyblok");
const posts = stories.map((story) => {
  return {
    title: story.data.content.title,
    date: new Date(story.data.published_at).toLocaleDateString("en-US", {
      dateStyle: "full",
    }),
    description: story.data.content.description,
    slug: story.data.full_slug,
  };
});

const { blok } = Astro.props;
---

<ul {...storyblokEditable(blok)}>
  {
    posts.map(
      (post: {
        date: unknown;
        slug: string | URL | null | undefined;
        title: unknown;
        description: unknown;
      }) => (
        <li>
          <time>{post.date}</time>
          <a href={post.slug}>{post.title}</a>
          <p>{post.description}</p>
        </li>
      )
    )
  }
</ul>

With the above code changes, we use getCollection("storyblok"); to load data from the Content Layer instead of using an API call. Since the Content Layer is populated just once in the beginning, we are saving a lot of API traffic.

To refresh the data store during development, you can use the s+enter hotkey in the terminal. The Storyblok loader is pre-optimized to load only delta updates. If you look at the source code for the loader, you will understand how it works:

...
  else if (storedLastUpdatedAt) {
    fetchParams.updated_at_gt = storedLastUpdatedAt;
  }
...
const stories = await storyblokApi?.getAll('cdn/stories', {
    ...fetchParams,
    ...storiesParams,
});

In the initial load we fetch all stories, but when you update the data store, the loader shall only fetch the stories which have been updated since the last load. If you fetch published stories then the delta updates rely on the published date instead of the updated.

We have successfully enabled delta updates in our setup now. Only content that changes will be fetched, thus saving the number of total requests sent to the Storyblok API.

Making the setup production-ready

In production, although the first build after you set up the content layer will fetch all stories, subsequent rebuilds will take full advantage of the setup, only re-fetching stories updated after the most recent build. Ensure you enable build caching in your build platform so the content layer works.

Before you use this setup in production, you should also ensure that you rebuild your deployed site once content changes in Storyblok. You can use Storyblok webhooks (opens in a new window) or the Tasks app (opens in a new window) to achieve this.

Handling deletion of stories:

When a story is deleted on Storyblok, it will continue to be cached in the content layer. You can fix this by triggering a webhook-based rebuild, but skip the build cache so that the content store is refreshed.

Caveats

There are some shortcomings in the current setup that you should be aware of :

• Localization: Currently, there is no special handling of stories that are multilingual alternatives to each other. This can be solved by maintaining a meta Content Layer to store these relationships.
• Missing Live Preview: Live Preview functionality won’t be available since we are using full static rendering.

Additional resources

• Content Layer: A Deep Dive
• Announcing a Storyblok Loader for the Astro Content Layer API