Continuously Write to a File Node

Tricks on writing & appending to a file in Node.js

This article covers the use of fs.appendFile and fs.writeFile functions, how they work in details. Specifically, we'll investigate them in a practical case.

Writing logs

Let's discover a use case where we want to write logs to a file. It seems like there is an obvious way to do this - call fs.writeFile each time we need it.

                          fs              .              writeFile              (              '              log.txt              '              ,              '              message              '              ,              '              utf8              '              ,              callback              );                      

The problem is writeFile replaces the file data each time we use the function, so we can't just write to a file. We could use a different approach: read a file data via fs.readFile, then append to the existing logs a necessary data and new line.

                          // we'll use callbacks in the article, but remember you always              //  can promisify those functions              // *we will not handle the errors in callbacks              const              newLogs              =              `              ${              Date              .              now              ()}              : new logs`              ;              fs              .              readFile              (              '              log.txt              '              ,              {              encoding              :              '              utf8              '              },              (              err              ,              data              )              =>              {              const              newData              =              data              +              newLogs              +              '              \n              '              ;              fs              .              writeFile              (              '              log.txt              '              ,              newData              ,              '              utf8              '              ,              callback              );              });                      

But this method also has cons. Each time we want to write new logs, the program opens a file, load all file data to memory, then opens the same file again and writes new data. Imagine how much resources a script will need in case of a large file.

Node has another method to do this simpler - fs.appendFile.

                          fs              .              appendFile              (              '              log.txt              '              ,              '              new logs              '              ,              '              utf8              '              ,              callback              );                      

This is much better, but what does this method do? Let's discover how appendFile is implemented.
lib/internal/fs/promises.js:

                          async              function              appendFile              (              path              ,              data              ,              options              )              {              // manipulations with the "options" argument, you can skip it              // up to the return statement              options              =              getOptions              (              options              ,              {              encoding              :              '              utf8              '              ,              mode              :              0o666              ,              flag              :              '              a              '              });              options              =              copyObject              (              options              );              options              .              flag              =              options              .              flag              ||              '              a              '              ;              return              writeFile              (              path              ,              data              ,              options              );              // so, writeFile anyway?              }              // ...              async              function              writeFile              (              path              ,              data              ,              options              )              {              options              =              getOptions              (              options              ,              {              encoding              :              '              utf8              '              ,              mode              :              0o666              ,              flag              :              '              w              '              });              const              flag              =              options              .              flag              ||              '              w              '              ;              // in our case, the "path" isn't a FileHandle, it's a string              if              (              path              instanceof              FileHandle              )              return              writeFileHandle              (              path              ,              data              ,              options              );              // "fd" is a file descriptor (FileHandle instance)              const              fd              =              await              open              (              path              ,              flag              ,              options              .              mode              );              return              writeFileHandle              (              fd              ,              data              ,              options              ).              finally              (              fd              .              close              );              }                      

We discover what is the FileHandle a bit further.

Still, the appendFile does pretty much the same as we did earlier. In details, it:

• opens a file, gets a file handle
• writes data to a file (calls 'write' which decide whether to write buffer or string(C++ bindings)).

Is it okay to write logs like that? Not really. It's okay for occasional writes, here's why.

appendFile opens a file each time we need to write logs. In some cases, it can cause EMFILE error which means an operating system denies us to open more files/sockets. For example, if we need to write a new log entry every 5ms, a Node script will open a file every 5ms. Also, you need to wait for the callback to make appendFile again, otherwise, the function will append a file data in a conflicting way. Example:

                          // Notice: `appendFile` is called asynchronously              fs              .              appendFile              (              '              log.txt              '              ,              '              1              '              ,              '              utf8              '              ,              callback              );              fs              .              appendFile              (              '              log.txt              '              ,              '              2              '              ,              '              utf8              '              ,              callback              );              fs              .              appendFile              (              '              log.txt              '              ,              '              3              '              ,              '              utf8              '              ,              callback              );              // log.txt can be as follows:              1              3              2                      

File descriptors

In short, file descriptor or file handle it's a reference to an opened file. They are non-negative integers. For example, standard input uses 0 value as a file handle, standard output uses 1, standard error output occupies 2 value. So, if you open a file programmatically, you'll get a file handle valued as 3 or more.
Node has its own wrapper for file handlers - FileHandle to perform basic operations on them (such as read, write, close, etc.).

The less opened file handles we have, the better. It means, fs.appendFile not a suitable solution to write logs.

Maybe streams?

Let's append to a file using writable streams:

                          // 'a' flag stands for 'append'              const              log              =              fs              .              createWriteStream              (              '              log.txt              '              ,              {              flags              :              '              a              '              });              // on new log entry ->              log              .              write              (              '              new entry              \n              '              );              // you can skip closing the stream if you want it to be opened while              // a program runs, then file handle will be closed              log              .              end              ();                      

What did we do here? We create a writable stream that opens log.txt in the background and queues writes to the file when it's ready. Pros:

• we don't load the whole file into RAM;
• we don't create new file descriptors each time a program writes to the file. The purpose of streams here is to write small chunks of data to a file instead of loading the whole file into memory.

Summaries

• Don't use fs.appendFile if you need to write to a file often.
• Use fs.appendFile for occasional writes.
• Don't use fs.writeFile (or fs.write) to write a large amount of data or when dealing with large files. Use writable streams instead.

Source

palmerints1937.blogspot.com

Source: https://dev.to/sergchr/tricks-on-writing-appending-to-a-file-in-node-1hik

Share this post