IndexedDB: Your all purpose browser cache

Working on a React static site that is pulling large amounts of data from a public API.

Problem

Specifically the API request for data looks like this <base url>/objects?metadataDate=YYYY-MM-DD Here the metadataDate indicates the data object has changed since the particular date specified and the date pattern is ISO 8601 format from above.

This returns for example a JSON object that looks like this

{
  "total": 44694,
  "objectIDs": [1, 2, 3, ...
  ]
}

This result can be quite large, but now you need to get the particulars of the metadata change from the object itself by individually sending the requests to another API endpoint structured like so <base url>/objects/<particular objectID>

Which returns a detailed JSON formatted object abbreviated format below

{
  "objectID": 1,
  "isHighlight": true,
  "accessionNumber": "50.25",
  "accessionYear": "1950"
  ...
}

This could generate thousands of requests for a particular search if you use a modified date that is in the past and each subsequent date after the first date will contain the data points of the latter searches. As in if you specified a metadataDate of 2024-05-01 and then specified in subsequent search a metadataDate of 2024-08-01 the latter search results would be completely contained within the former so there is no need to request the details of the common objects.

What is needed here is a cache that operates within the browser that is uniquely identifiable and is going to be stable per interaction session and reduce the request/response flow need by using caching within the browser to preserve already obtained API data.

Solution

Use the IndexedDB within modern browsers to create a key/value lookup table. It is a modern large storage relational database included in modern browsers with async/await support. Think of it as localStorage but x 1,000,000.

First step

Building off of the idb project async wrappers to create a get/set quick cache of any data in TypeScript

import {deleteDB, IDBPDatabase, openDB} from 'idb';

export const dbSettings = Object.freeze({
  Version: 1,
  Name: "<name your cache here>",
  KeyStore: "<add the table name here>"
});

async function getDB() : Promise<IDBPDatabase<unknown>>    {
  return await openDB(dbSettings.Name, dbSettings.Version, {
    upgrade(db) {
      db.createObjectStore(dbSettings.KeyStore, {autoIncrement: true});
    }
  });
}

export async function set(id: number, inputValue: unknown) : Promise<IDBValidKey | undefined> {
  try {
    const db = await getDB();    
    const tx = db.transaction(dbSettings.KeyStore, 'readwrite');
    const store = tx.objectStore(dbSettings.KeyStore);
    const addResult = await store.put(inputValue, id);
    await tx.done;
    db.close();
    return addResult;
  } catch (exc) {
    console.trace(`Failed dataStore.set ${exc}`);
  }
  return undefined;
}

export async function get(id: number) : Promise<unknown> {
  try {
    const db = await getDB();
    const tx = db.transaction(dbSettings.KeyStore, 'readonly');
    const store = tx.objectStore(dbSettings.KeyStore);
    const findResult = await store.get(id);
    await tx.done;
    db.close();
    return findResult;
  } catch (exc) {
    console.trace(`Failed dataStore.get ${exc}`);
  }
  return null;
}

export async function initializeCache(): Promise<void> {
  await deleteDB(dbSettings.Name);
}

Example use

Format is (key, data) on the call as in something like this to save a value into the cache await set(objectId, newItem);

Detailed use below

import {get, set} from "./Cache.ts";

export async function getItemDetails(objectId: number, testMode: boolean = false) {
  if (testMode) {
    return testValues.item;
  }
  const cachedResult = await get(objectId) as ItemDetails;
  if (cachedResult !== undefined && cachedResult !== null) {
    return cachedResult;
  }
  const response = await fetch(`${API_V1_BASE_URL}/${objectId}`, {signal: requestSignal});
  const itemData = await response.json();
  const newItem: ItemDetails = {...itemData};
  await set(objectId, newItem);
  return newItem;
}

And now you can have one liner access to an async, relational, private per site database that can store all the relevant and requested data for as long as you want or need and make instantaneous data requests instead of overloading or getting rate limited on an API that operates in this manner.