Basics of stderr and stdout on Ruby scripts Oct 23 2018

In our scripts, we present information to our users via the two standard output streams: standard output and standard error. In this post, I’ll explain why it is useful to distinguish between standard output and standard error, and share tips on how to interact with the streams when our script’s output is being redirected or used on a non-interactive shell.

The first part is the basics if you already know the basics you can skip to the last section to see an interesting example on how to take advantage of the streams.

Bash Beyond Basics Increase your efficiency and understanding of the shell

If you are interested in this topic you might enjoy my course Bash Byond Basics. This course helps you level up your bash skills. This is not a course on shell-scripting, is a course on improving your efficiency by showing you the features of bash that are seldom discussed and often ignored.

Every day you spend many hours working in the shell, every little improvement in your worklflows will pay dividends many fold!

Learn more

Streams basics

In Ruby, we can access the standard output and standard error streams by the constants STDOUT and STDERR. Normally when we want to print to the screen we only use puts, in reality, this is syntactic sugar for STDOUT.puts. As with any other IO stream, we can send content to the stream using puts, print, printf and the other commands provided by the IO class. For example:

1
2
3
4
#!/usr/bin/env ruby

STDOUT.puts('Hello, from STDOUT') #this will go to standard output
STDERR.puts('Hello, from STDERR') # this will go to standard error

If we run this script we will get on the screen the following:

1
2
3
$ ./streams_script
Hello, from STDOUT
Hello, from STDERR

It starts to get interesting when we redirect the output of one of the streams, in BASH the file identifier 1 is assigned to standard output and the file identifier 2 is assigned to standard error, so we could run something like this:

1
2
$ ./streams_script 2> std_error.log
Hello, from STDOUT

And the message from STDERR won’t show up, it has been redirected to the file std_err.log. We can do the same with standard output.

1
2
$ ./streams_script 1> std_out.log
Hello, from STDERR

Now the message from STDOUT won’t show up on the screen because it was redirected to the file std_out.log. We could redirect standard output to one file and standard error to another file.

1
$ ./streams_script 1> std_out.log 2> std_error.log

Or redirect both standard output and standard error to the same file:

1
$ ./streams_script &> out.log

There is more to redirection but this should give you a good idea.

Why is this useful?

When we send everything to standard output, even messages we want the user to take note of, we prevent our user from taking advantage of stream redirection. But when we allow the user to have access to those two streams the user can make decisions on where to look for data and has the ability to redirect to wherever he likes.

Let’s look at an example, imagine we have a script that processes files and if it encounters an error it’ll print a message to standard output telling the user that there was an error. The user runs the script and the screen starts scrolling with messages of the files that it is processing, at some point the user sees an error and writes a note to check what was wrong with that file. For example:

1
2
3
4
5
6
7
8
$ ./ruby_file_processing_script 
Start processing
Processing file 1 - ok
Processing file 2 - ok
Processing file 3 - ok
Processing file 4 - error
Processing file 5 - ok
Done processing

This isn’t a problem if the user is only processing a few files, but what happens if he wants to leave the process running through the night and it’ll process some 10,000 files, in the morning the user will have to scroll through the screen and try to find errors reported on the screen.

If you instead create your script to separate the streams, the user might do something like the following:

1
$ ./ruby_file_processing_script 2> error.log

This redirection works better than trying to read all the lines printed on the screen by the script. The user can check this file in the morning and have an easier time correcting the errors. This is the type of scripts we would like to build for our users, useful, flexible and easy to integrate to any workflow.

More tips for redirection and checking for interactive shell

We have covered the basics of redirection, now let’s move to something more interesting. Let’s check if the script is running using an interactive shell. Scripts can be run on an interactive, when the user is logged in to a shell and runs the script directly on the shell, that shell session is interactive, our script can interact “interactively” with the user. If our script is being run on a background process or by other script or the output is being redirected it is said to be running a non-interactive TTY. There is more to it but, again this should give you a basic idea, if you want to read more you can read this article on The Linux Documentation Project , or if you want me to write about interactive and non-interactive shells let me know.

Now how can we take advantage of the fact that the user is on an interactive shell?

We could display a progress bar to the user using the standard error if the user is on an interactive shell and if the user is not on an interactive shell we can just display messages that would be useful for the user to log.

Let’s look at an example script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#!/usr/bin/env ruby

total_items = 70
error_files = []
1.upto(total_items) do |i|
  error_files << i if Random.rand(7) == 1
  if STDERR.tty?
    percentage = (i.to_f / total_items) * 10
    STDERR.printf("\rprocessing: [%-10s] - Processing item: %s", '=' * percentage, i)
  else
    STDOUT.printf("Processing item #{i}\n")
    STDERR.printf("Error Processing item #{i}\n") if Random.rand(7) == 1
  end
  sleep(0.1)
end
complete_message = "\rDONE - #{total_items} processed"
if STDERR.tty?
  STDERR.printf("\r%-100s\n",complete_message)
  STDERR.puts("Error on the following files: #{error_files}") if !error_files.empty?
else
  STDOUT.puts(complete_message)
end

We use STDERR.tty? to check if we are on an interactive shell (TTY) and based on that validation we make the decision of how our script should behave. I added some random number generation to simulate files with errors, they are just to exemplify the behaviour.

If you run the script on your shell you’ll see something like the following:

1
2
3
$ ./ptty
DONE - 70 processed
Error on the following files: [10, 11, 14, 19, 21, 26, 29, 34, 51, 63]

And while it was running you should have seen a progress bar. And now if you run it redirecting the standard error to a log file, you will see something like the following:

1
2
3
4
5
6
7
8
./ptty 2> error.log
Processing item 1
Processing item 2
Processing item 3
...
Processing item 69
Processing item 70
DONE - 70 processed

And if you check the error.log file you should get a list of the files that had the fictitious errors, you should see something similar to this:

1
2
3
4
5
6
7
8
9
10
11
$ cat error.log
Error Processing item 1
Error Processing item 3
Error Processing item 21
Error Processing item 26
Error Processing item 29
Error Processing item 46
Error Processing item 48
Error Processing item 50
Error Processing item 52
Error Processing item 65

That is some useful behaviour! If the user runs the script on an interactive terminal we can present a nice progress bar but if he is trying to capture the errors let’s give the user data that is easier to process or that might even be redirected to another script.

I hope this post gave you some ideas on how to improve the user experience on your scripts, command line interfaces should be easy to integrate to other workflows, but that doesn’t mean we should present everything just thinking of computers and programs thinking too of enhancing your user experience.

Let me know what you think and if you have any comments/corrections/asks of this or other posts let me know I’ll be glad to get back to you and if you would like me to write about a specific topic send me a message and I’ll see what can I do.