Wond3rBoi Wond3rBoi - 3 years ago 209
Node.js Question

Why can't I pipe npm install to sed, while preserving color and progress updates?

If I type

npm i | sed "s/^/ /"
, the output of
npm i
is not spaced over when printed to stdout. E.g. I get the following:

$ npm i | sed "s/^/ /"
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0
(node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for
fsevents@1.1.2: wanted {"os":"darwin","arch":"any"} (current:
{"os":"linux","arch":"x64"})
npm WARN [name_removed]@0.0.2 No repository field.
npm WARN [name_removed]@0.0.2 No license field.


instead of:

$ npm i | sed "s/^/ /"
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0
(node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for
fsevents@1.1.2: wanted {"os":"darwin","arch":"any"} (current:
{"os":"linux","arch":"x64"})
npm WARN [name_removed]@0.0.2 No repository field.
npm WARN [name_removed]@0.0.2 No license field.


Edit:
The warnings are going to stderr (duh...), so i need to use
npm i 2>&1 | sed 's/^/ /
, but this removes colors from the output and I don't see the progress bar, which you can see in the gif below.

enter image description here
enter image description here

Edit2: Color is fixed via
npm i --color=always | sed 's/^/ /
, but i still don't see the progress bar. In addition, it seems to add lines to the line following all the output...I assume this is caused by a color code being output? You can see this phenomenon in the gif below:

enter image description here

Current status:
enter image description here

Answer Source

You're experiencing several issues here, most of them related to the way npm handles its output.

Catching warnings

Note first that npm outputs warnings and updates progress on stderr, while only final results go to stdout. So, in order to process warning, you'll have to redirect stderr to stdout with something like:

npm install 2>&1 | sed 's/^/    /'

Preserving color

But now with stderr piped to sed process, you'll notice npm omits coloring! This, however, is a standard behaviour for most command line tools (like ls, grep, etc.). They'll output ANSI escape (color) sequences only when the output is going to a TTY device (i.e. to user, not file, or pipe). The usual way of determining if a file descriptor is connected to a TTY is via isatty(int fd) function. It turns out (after some digging) npm is using the same mechanism.

To fix coloring problem, we have two options:

(1) We can force color output with --color=always option (similar to ls, grep, and others):

npm install --color=always 2>&1 | sed 's/^/    /'

or, (2) we can use a tool called script which will fake a TTY output device for any program/script run:

script -feqc 'npm install' /dev/null | sed 's/^/    /'

Note we don't have to redirect stderr to stdout anymore, script does that for us. Also, script will save the complete output to a file of our choosing (in this case we don't need it, so we're saying /dev/null). (Btw, -f will flush output after each write, -e will ensure the exit code of the command is returned to parent/shell, -q forces quiet mode with no info messages, and with -c cmd we provide the command to run.)

Preserving progress updates

Ok, so now we have warnings indented and color preserved, but we've lost our progress (bar) updates!

Why has this happened? Well, because npm outputs the complete progress in a single line. On each progress update it will move to character position zero (of the same line!) and print the new progress. For sed, the complete progress is just a single line, and since sed is line-oriented, it waits until the end of the line (\n) before any processing (and output).

Obviously, we need to go one level lower - and process character by character. To achieve the effect of indentation, we'll replace each occurrence of \n with \n<4 spaces>.

Usually, for character translation we can use tr, but here we need more than that, since in some cases (\n) we need to expand one character to several. One way to do it is with this simple bash script/function:

#!/bin/bash
# read each character of stdin, indenting each line
interactive_indent() {
    local space='    '
    echo -n "$space"
    while IFS= read -r -d '' -n1 chr; do
        [[ $chr == $'\n' ]] && chr="\\n\\r$space"
        [[ $chr == $'\r' ]] && chr="\\r$space"
        echo -ne "$chr"
    done
    echo -ne '\r'
}

For example:

$ echo -e 'one\ntwo\rthree' | interactive_indent
    one
    three

Finally, we come to the solution for our interactive npm process:

script -feqc 'npm install' /dev/null | interactive_indent

This will pass-through each character (displaying progress), while indenting each line (after \n or \r).

Note that our interactive_indent function is a little more complex than a simple \n-to-\n<spaces> replacer. We also had to handle carriage returns (\r) which are heavily used by npm for both progress update and dependency tree drawing, and ensure each new line starts at position zero (hence the \r alongside each \n).

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download