Caching a Read in File in Javascript

Caching results

Caching is a powerful way to do a procedure only once and thus speed up an application. For instance, generate images from a PDF yields the aforementioned result for the same input file, so there is no need to run the costly process from scratch every time. Saving previous results and reusing them when advisable can take an application that takes ages to run and make information technology a quick one.

Caching in memory is a good starting place. Information technology works by keeping the previous results in a variable and so that it'southward available the next fourth dimension a costly process runs. Only every bit memory is cleared when the procedure exits, it can not reuse results betwixt restarts.

File-based caching is a practiced solution for this. Files are persistent beyond restarts, providing a durable identify to store results. But they too come with an extra set of problems.

Structure

All file-based caching follows this full general construction:

                          const              cacheDir              =              // where to put cache files              const              cacheKey              =              // summate cache central for the input              const              cacheFile              =              path              .              join              (              cacheDir              ,              cacheKey              );              if              (              exists              (              cacheFile              ))              {              // the consequence is cached              return              fs              .              readFile              (              cacheFile              );              }              else              {              // calculate the outcome and store it              const              result              =              // run the process              await              fs              .              writeFile              (              cacheFile              ,              result              );              render              outcome              ;              }                      

It calculates the cache key and the cache directory, and then checks if at that place is a file in that place. If there is, information technology reads the contents (cache hit), if there is none, then it calculates the issue then writes the cache file (enshroud miss).

Let'southward break downward each role!

Enshroud directory

The first question is: where to store the enshroud files? A skilful cache directory is excluded from version command, and information technology is removed from time-to-fourth dimension.

In that location is an effort to standardize a persistent cache location for Node.js applications in node_modules/.enshroud. It has an advantage over /tmp that it survives machine restarts, while it is in the node_modules directory that is unremarkably recreatable using the package-lock.json.

The notice-enshroud-dir package provides an easy-to-employ way to locate the cache directory.

To initialize and get the cache directory, employ this lawmaking:

                          const              findCacheDir              =              require              (              "              notice-cache-dir              "              );              const              {              promises              :              fs              ,              constants              }              =              crave              (              "              fs              "              );              const              getCacheDir              =              (()              =>              {              const              cacheDir              =              thunk              ();              permit              prom              =              undefined              ;              return              ()              =>              prom              =              (              prom              ||              (              async              ()              =>              {              await              fs              .              mkdir              (              cacheDir              ,              {              recursive              :              true              });              return              cacheDir              ;              })());              })();                      

This uses the async lazy initializer pattern to create the directory merely when needed.

Related

How to run something asynchronous only once and when needed

Cache key

All caching depends on a good cache key. It must exist known before running the calculation and must exist unlike when the output is different. And, of course, should be the aforementioned when the output is the same.

I found information technology a best do to hash the parts before concatenation and so hash the result again. Since hashing makes a fixed-length string, it is resistant to concatenation problems (such as "ab" + "c" === "a" + "bc").

                          const              crypto              =              require              (              "              crypto              "              );              const              sha              =              (              x              )              =>              crypto              .              createHash              (              "              sha256              "              ).              update              (              ten              ).              digest              (              "              hex              "              );                      

What should be in the cache key? The input information is an obvious candidate, only unlike memory-based caching, some descriptor of the process should also be included. This is to make sure that new versions of the packages invalidate the caches.

For case, when I needed to enshroud the results of a PDF-to-images process, I needed to become the version of the external programme that did the calculations (pdftocairo). It provides a version() call that calls the procedure with the -5 flag to print its version.

Simply non only the external program influences the result but also the Node.js packet. Its version is in the parcel.json.

The getVersionHash() function returns the hash of these versions:

                          const              pjson              =              crave              (              "              ./package.json              "              );              const              {              version              }              =              crave              (              "              node-pdftocairo              "              );              const              getVersionHash              =              (()              =>              {              let              prom              =              undefined              ;              return              ()              =>              prom              =              (              prom              ||              (              async              ()              =>              sha              (              sha              (              await              version              ())              +              sha              (              pjson              .              version              )))());              })();                      

The enshroud primal is the version hash and the source hash: sha(expect getVersionHash() + sha(source)).

Cache file

The enshroud file is the enshroud directory and the cache key:

                          // source is the input              const              cacheFile              =              path              .              bring together              (              await              getCacheDir              (),              sha              (              await              getVersionHash              ()              +              sha              (              source              )));                      

Handle caches

Beginning, the enshroud logic needs to determine whether the result is cached or not. This is a check whether the file exists or not:

                          const              fileExists              =              async              (              file              )              =>              {              try              {              expect              fs              .              access              (              file              ,              constants              .              F_OK              );              return              true              ;              }              grab              (              e              )              {              render              faux              ;              }              };              if              (              await              fileExists              (              cacheFile              ))              {              // read and return              }              else              {              // summate and write              }                      

If the result is a single file or value, it's easy to handle the two cases:

                          if              (              fileExists              (              cacheFile              ))              {              // the issue is cached              render              fs              .              readFile              (              cacheFile              );              }              else              {              // summate the upshot and shop it              const              result              =              // run the process              look              fs              .              writeFile              (              cacheFile              ,              result              );              return              result              ;              }                      

Cache multiple files

Storing multiple results is also possible, just zip what you desire to cache and write the archive to the cache. I prefer the JSZip library to handle archiving in Javascript:

                          const              JSZip              =              require              (              "              jszip              "              );              const              stream              =              crave              (              "              stream              "              );              const              util              =              require              (              "              util              "              );              const              finished              =              util              .              promisify              (              stream              .              finished              );              if              (              wait              fileExists              (              cacheFile              ))              {              const              file              =              await              fs              .              readFile              (              cacheFile              );              const              aught              =              await              JSZip              .              loadAsync              (              file              );              const              files              =              expect              Promise              .              all              (              Object              .              values              (              zip              .              files              )              // to make sure the outcome array contains the files in the same ordering              .              sort              (({              name              :              name1              },              {              name              :              name2              })              =>              new              Intl              .              Collator              (              undefined              ,              {              numeric              :              true              }).              compare              (              name1              ,              name2              ))              .              map              ((              file              )              =>              file              .              async              (              "              nodebuffer              "              ))              );              return              files              ;              }              else              {              const              res              =              // calculate the result files              const              zip              =              new              JSZip              ();              res              .              forEach              ((              file              ,              i              )              =>              {              zip              .              file              (              String              (              i              ),              file              );              });              await              finished              (              nix              .              generateNodeStream              ({              streamFiles              :              true              })              .              pipe              (              createWriteStream              (              cacheFile              ))              );              return              res              ;              }                      

With this solution, whatever number of files can be cached in a single zippo file.

Decision

File-based caching is a powerful tool to speed up applications. Only information technology besides makes cache-related errors to survive restarts, so extra care is necessary when implementing it.

Caching a Read in File in Javascript

Source: https://advancedweb.hu/how-to-implement-a-persistent-file-based-cache-in-node-js/

Share this post