FunctionPackage: exclToCDocOverviewCGDocRelNotesFAQIndexPermutedIndex
Allegro CL version 9.0
Moderate update since 9.0 release.
8.2 version

run-shell-command

Arguments: command &key input output error-output separate-streams (wait t) if-input-does-not-exist if-output-exists if-error-output-exists show-window environment directory uid gid effective initgroups-user (share-open-files t)

Introductory comments

The :osi module (see Operating System Interface Functionality in os-interface.htm) has these new operators relating to running subprocesses: the function command-output and the macros with-command-output and with-command-io. They are higher-level than run-shell-command and shell and are now recommended when the interaction with the subprocess requires input or produces output that must be captured. The new operators do not have separate description pages. They are described in OSI process/uid/gid interface functions in os-interface.htm.

This function, originally written and named for Unix machines, is badly misnamed for Windows, because on Windows, you can only run programs. Shell commands, such as dir, are not acceptable as a value of the command argument. See below in this entry for discussion on this point.

Also note that there is often a Lisp tool equivalent to a command. (For example, the Lisp function directory can be used in place of dir on Windows and ls on Unix. See below under the heading run-shell-command and Windows for why "dir" cannot be the value of the command argument.) It is always preferable to use Lisp tools where possible.

This is particularly true in an application delivered to others. The action of spawning a process, which might involve spawning a shell, may have consequences on another user's machine which you (the application developer) do not expect, and these consequences may interfere with the operation of your application in ways that are difficult to debug.

This is a powerful but complicated function, which has many options and modes of behavior. We have tried to provide sufficient internal links to navigate about this description. All necessary information is here somewhere, so if you scroll done you will likely find what you need.

0. Beginning of function documentation: input to and output from the command and return values

The process spawned by this function will generally have (always on UNIX, console apps on Windows) standard input, standard output, and standard error. If you want input beyond what is part of the command argument, you need to get a handle on the standard input. If you want to capture (and possibly redirect) the output or the error output, you need to get a handle on standard output and standard error.

You can, as we describe in the discussion of the input, output, and error-output keyword arguments below, set things up yourself, by specifying streams or file pathnames. And, when the wait keyword argument is nil, you can have run-shell-command set up Lisp streams to connect to process's input, output, and error-output. In that case, the created streams are among the multiple values returned by run-shell-command, as we describe next in the discussion of return values.

This function returns either one value when the wait argument is true (it defaults to t), or three or four values when wait is nil. When wait is nil, the number of values depends on the value of the separate-streams argument (described below).

When commands display information (such as ls displaying file and directory information), that information is not what is returned by run-shell-command. When wait is true (the default), the single return value is the exit status of the spawned process. When wait is nil, the multiple return values are 2 or 3 values which can be streams or nil and a final value which is the pid (process id number) of the spawned process. Output from the command is sent to whatever the standard out of the spawned process is connected to.

1A. Return value when :wait is t (the default)

When run-shell-command is called with the wait keyword argument is true (or unspecified since the default is t), then the single return value is the exit status of the spawned process. You must set up handling of the spawned process's input, output, and error-output using the input, output, and error-output keyword arguments as described below. Here are two examples showing what we mean. These examples use printenv and so do not work on Windows, but they do illustrate thr return value.

;; The command is 'printenv DISPLAY'. First we call it with
;; :wait t (just for emphasis, :wait defaults to t):

cl-user(131): (run-shell-command "printenv DISPLAY" :wait t)
192.132.95.16:12.0
0

;; The DISPLAY value is printed BUT NOT RETURNED. What is returned
;; is the single value 0, the exit status of the call to printenv.
;; This is made clear by the following:

cl-user(133): (multiple-value-list (run-shell-command "printenv DISPLAY"))
192.132.95.16:12.0
(0)

;; Here we let :wait simply default to t. Again the DISPLAY value is
;; printed, but, as trhe singleton list of return values -- (0) -- shows,
;; not returned. The output is being sent to the standard out of the Lisp
;; process, which the spawned process inherits. 

;; In our final example, we send the process output to a stream open to 
;; a file:

cl-user(136): (with-open-file (s "test.txt" :direction :output
			       :if-exists :supersede)
		(run-shell-command "printenv DISPLAY" :output s))
0

;; And here are the contents of the file:

cl-user(137): (run-shell-command "cat test.txt")
192.132.95.16:12.0
0
cl-user(138): 

1B. Return values when :wait is nil

If you specify :wait nil, be sure to also see More on the :wait argument below for information on reaping the process when it completes.

Because the process has not completed when run-shell-command returns, its exit status is not known and thus not returned. The exit status is returned by reap-os-subprocess when that function is called. Again, see More on the :wait argument below.

When wait is specified nil, run-shell-command returns three or four values: three when the separate-streams keyword argument is nil, four when separate-streams is true. The separate-streams argument is discussed in this section and in its own section below).

When the value of the wait keyword argument is specified nil, then you can have run-shell-command create streams for any or all of the spawned process's standard input, output, and error-output. Any created streams must be made available, and this is done by returning them. (run-shell-command returns immediately when wait is nil, and so any returned streams are available at once.)

The interface is complex and admittedly non-intuitive (but it has been around so long that it cannot be easily changed). Just keep in mind that multiple values are returned, that the last return value is the pid of the spawned process's pid, and that the earlier return values are either streams set up by run-shell-command or nil.

The streams that run-shell-command might set up are individual streams for the spawned process's standard in and out, a bidirectional stream for both standard in and out, and/or an individual stream for the spawned process's error output.

run-shell-command will set up a stream for standard input when input is :stream, for standard output when output is :stream, and for error output when error-output is :stream. If both input and output are :stream, a single bidirection stream is set up for both unless the separate-streams keyword argument (described below) is specified true, in which case different streams are set up for input and output. run-shell-command will not itself set up a stream for input, output, or error output when, respectively, input, output, and error-output have any value other than :stream. The other allowable values for those arguments are discussed below.

Here are some sample calling templates to illustrate the range of possible return values. Unspecified keyword arguments (indicated by suspension points -- ...) do not affect the indicated return values.

;;  First a reminder about calls with :wait t.
;;  When the :wait argument is true (as it is by default), 
;;  a single value is returned. Here are two examples with :wait t.
;;  The second signals an error since none of :input, :output, or
;;  :error-output can be :stream when :wait is true.

(run-shell-command command :wait t ...)
 RETURNS as a single value the exit status of the spawned
    process which executes command (note that the default value
    of :wait is T so ':wait t' need not be specified)

(run-shell-command command :wait t 
                   :input :stream
                   :output :stream
                   :error-output :stream
                   ...)
 ERROR: none of :input, :output, or :error-output can be :stream
    when :wait is t.

In all remaining examples, wait is nil and multiple values are returned. input, output, and error can each have the value :stream or other values described below. For showing what is returned, it only matters whether each of the arguments input, output, and error is :stream or not, so we show the value as :stream, or [some value other than :stream], or [any value including :stream].

run-shell-command, when wait is nil, returns three values when the separate-streams keyword argument is nil (the default), and returns four values when separate-streams is true. separate-streams refers to streams for input and output: when nil, a single stream is set up for input, or output, or both (a bidirectional stream in that case; when true, separate streams are set up. The :separate-streams argument is described below.

;; separate-streams is nil in the first set of examples:

(run-shell-command command 
                   :input [some value other than :stream]
                   :output [some value other than :stream]
                   :error-output [some value other than :stream]
                   :separate-streams nil :wait nil ...)
 RETURNS three values
 nil
 nil
 the process id of the spawned process

(run-shell-command command 
                   :input :stream 
                   :output [some value other than :stream]
                   :error-output [some value other than :stream]
                   :separate-streams nil :wait nil ...)
 RETURNS three values
 a stream (which acts as standard input to the spawned process)
 nil
 the process id of the spawned process

(run-shell-command command 
                   :input [some value other than :stream]
                   :output :stream
                   :error-output [some value other than :stream]
                   :separate-streams nil :wait nil ...)
 RETURNS three values
 a stream (which acts as standard output from the spawned process)
 nil
 the process id of the spawned process

(run-shell-command command 
                   :input :stream 
                   :output :stream
                   :error-output [some value other than :stream]
                   :separate-streams nil :wait nil ...)
 RETURNS three values
 bi-directional-stream (standard in and standard out of spawned process)
 nil
 the process id of the spawned process

(run-shell-command command 
                   :input [any value including :stream]
                   :output [any value including :stream]
                   :error-output :stream
                   :separate-streams nil :wait nil ...)
 RETURNS three values:
 stream-or-nil (stream if either :input or :output is :stream)
 stream (for standard error of the spawned process)
 the process id of the spawned process

;;  In the next example, separate-streams is
;;  t and four values are returned.

(run-shell-command command :separate-streams t :wait nil ...)
 RETURNS four values:
 stream-or-nil (a stream to standard input if :input is :stream, nil otherwise)
 stream-or-nil (a stream to standard output if :output is :stream, nil otherwise)
 stream-or-nil (a stream to error-output if :error-output is :stream, nil otherwise)
 the process id of the spawned process

;; In this next example, we create a stream ourselves and use it for
;; :output. Note that RUN-SHELL-COMMAND returns NIL for the output stream
;; value (second returned value since separate-streams is true)
;; even though output is actually a stream, since RUN-SHELL-COMMAND
;; itself set up the stream:

cl-user(139): (with-open-file (s "test.txt" :direction :output
			       :if-exists :supersede)
		(run-shell-command "printenv DISPLAY" :output s :wait nil
				   :separate-streams t))
nil
nil
nil
13147
cl-user(140): 

2. The command argument

The command argument can be a string containing a shell command (in Unix) or a program (in Windows). On Unix only, command can be a simple vector (element type t).

Here are two Unix invocations, one using a vector and one a string:

(run-shell-command #("ls" "ls" "-l" "dcl"))
(run-shell-command "ls -l dcl") 

Note that when command is a string, and wait is nil, the returned process-id will be that of the shell, not the command itself, unless the command is prefaced with the shell `exec' built-in command. Thus

(run-shell-command "ls" :wait nil)

will return the process-id of the shell in which ls is run while

(run-shell-command "exec ls" :wait nil)

will return the process-id of ls as will

(run-shell-command #("ls" "ls") :wait nil)

3. input, output, and error-output keyword arguments

The values of input, output, and error-output keyword arguments control what the spawned process will use as standard input (file descriptor 0), standard output (file descriptor 1) and standard error (file descriptor 2). The values can be

error-output has an additional allowed value: :output, which directs standard error to the same place as standard output (file descriptor 1 rather than 2).

The default value in all cases is nil.

4. The separate-streams keyword argument

The separate-streams keyword argument causes separate streams to be created and returned (rather than a single, bi-directional stream) when input and output are both :stream. When separate-streams is true, four values are returned, the first three are the input stream, the output stream, and the error-output stream, and the fourth is the process id.

In all cases, nil will be returned in place of a stream if the relevant argument is not :stream (see the call templates described above).

The value of separate-streams only matters when the value of the wait is nil.

5. The directory keyword argument

The directory keyword argument can be used to specify the directory in which the command runs. It defaults to nil, which results in the command being run in the directory returned by current-directory. If its value is a directory pathname or namestring, the command is run in that directory.

6. More on the :wait argument

wait may be t or nil. If wait is t, Lisp will wait for the command to exit before resuming. If wait is nil, Lisp will start the process and then resume without waiting for it to finish. The default for wait is t.

Note that if wait is specified as nil, then the process will remain in the system after it completes until either Lisp exits or Lisp executes reap-os-subprocess to inquire about the exit status. To prevent the system becoming clogged with processes, a program that spawns a number of processes with :wait set to nil must be sure to call reap-os-subprocess after each process finishes. Further, the streams returned by run-shell-command must be closed (the system will not close them automatically).

Also see Killing a process started with :wait nil below.

7. The :if-* arguments

The keyword arguments if-input-does-not-exist, if-output-exists and if-error-output-exists are used when input, output or error-output are pathnames (or strings naming files).

Lisp uses open to open a stream to the file identified by the pathname (or string) and the values of if-input-does-not-exist, if-output-exists and if-error-output-exists are passed to the open function as the value of its if-does-not-exist parameter (for input) and the if-exists parameter (for output and error-output.)

The permissible values for if-does-not-exist are :error, :create and nil.

Those for if-output-exists and if-error-output-exists are :error, :overwrite, :append, :supersede and nil. The default is :error in all cases.

8. The environment keyword argument

The value of environment should be an association list of names and values or a list of lists, where the name is the first element of the list and the value is the second (further elements are ignored). Names and values should be strings. Each name should be the name of an environment variable to set in the process being spawned, and the corresponding value should be the desired value for that variable. The environment variable is only set in the process being spawned, and is not set in the process executing the run-shell-command, nor is it remembered in future calls to run-shell-command.

run-shell-command adds the specified environment variables and values to the existing environment and passes that to the subprocess it is starting. Specifying an environment name that already exists will cause that name's value to be replaced with the new value specified.

4096 bytes is allocated for storing new names and values.

In the following example, we show that the DISPLAY environment variable has a value, but that value is changed in the spawned process when we specify :environment '(("DISPLAY" "111.222.33.444:0")), and that the change is not remembered.

cl-user(6): (run-shell-command "printenv DISPLAY")
192.132.95.213:0
0
cl-user(7): (run-shell-command "printenv DISPLAY"
                               :environment 
                               '(("DISPLAY" "111.222.33.444:0")))
111.222.33.444:0
0
cl-user(8): (run-shell-command "printenv DISPLAY")
192.132.95.213:0
0
cl-user(9): 

9. run-shell-command and Windows

On all Windows OS's, run-shell-command executes programs but does not invoke shell commands. The function is therefore misnamed for Windows. It is called run-shell-command to provide cross-platform compatibility between Windows and Unix. It does behave differently on Windows and Unix. The difference is related to differences between UNIX and Windows.

On Unix run-shell-command spawns a Unix process and runs the executable program or Unix shell command provided as argument to the function. On Windows, the argument to run-shell-command command is started directly and so works with programs but the Windows command shell is not invoked and so run-shell-command  does not work with shell commands. An example of a program is Notepad. Either of these forms will start Notepad on Windows:

(run-shell-command "notepad") 
(run-shell-command "notepad.exe") 

To run a DOS shell command, the argument to run-shell-command has to start the shell and tell it to run the command.

Here is an example:

On Windows the name of the shell is cmd.exe. Here is an example of using run-shell-command on Windows:

(run-shell-command "cmd /c start \"c:\\Program Files\\\"") 

which opens a command prompt in the c:\\Program Files\\ directory.

This displays the Windows explorer and avoids cmd altogether:

(run-shell-command "explorer c:\\acl")

Also:

10. The :show-window keyword argument (Windows only)

The show-window keyword argument only has effect on Windows. The value controls how the window created by the program run by run-shell-command first appears. The value can be an integer. The integer should be the value of one of the SW_ constants defined in the winuser.h include file that is part of the Windows SDK. The symbolic value should be preferred over an integer, however, since this will be portable in the face of changes to the Windows SDK header files and use on other operating systems. The value can also be one of the following symbols:

11. The :share-open-files keyword argument (Windows only)

When share-open-files is true (the default value), behavior is as it has been before this argument was added: all open file handles that could be shared will be shared.

When share-open-files is specified as nil, then the input, output, and error-output keyword arguments must also be nil (which is their default). In this case no open file handles will be shared and the spawned process will open its own standard io handles if it needs them.

12. Threads and streams on Windows

On Windows, Allegro CL starts an operating system thread for each output stream created, to be a transfer agent for the data. This thread cannot exit until it knows all the data has been transferred from the external shell process to the Lisp world. This will happen if the external process finishes and the agent has read all the data it produced, or if the stream is explicitly closed on the Lisp side. Since the Lisp portion of the stream has a limited buffer space, there may well be data still to be read long after the external process finishes; the agent cannot transfer all the data to the Lisp buffer until Lisp activity makes room for it by reading the earlier data. The Lisp application that uses run-shell-command with stream output should explicitly close any stream it does not read to the end, or it risks finding the available pool of OS threads completely taken up by zombie stream agents.

13. run-shell-command and Unix

run-shell-command starts a process with execlp() when command is a string and (as we said above) execvp() when command is a simple vector.

The SHELL environment variable is used to determine the type of shell spawned. If SHELL has no value. /bin/csh is used. If that fails, /bin/sh is used.

14. The environment in which run-shell-command runs its processes

run-shell-command starts up subprocesses of the Lisp process. In order to modify an environment variable for these subprocesses, you must modify the environment for the Lisp process -- it does no good to modify the environment for the parent process of the Lisp process (i.e. the shell where Lisp was started, or Emacs, if Lisp is started as a subprocess of Emacs).

You can poll environment variables with getenv, and you can set environment variables for the Lisp process, and thus for subprocesses created by Lisp (by run-shell-command, e.g.) with setf and getenv.

15. The uid, gid, effective, and initgroups-user keyword arguments

These arguments only have meaning on UNIX platforms. They are not supported on Windows.

uid and gid are numbers representing user and group ids. effective is a boolean which indicates that uid and gid are effective user and group ids. initgroups-user is a string naming a user.

Here is an example using command-output (which also has the new arguments):

cl-user(1): (require :osi)
t
cl-user(2): (excl.osi:command-output "whoami" :uid 483)
("layer")
nil
0
cl-user(3): (excl.osi:command-output "whoami")
("root")
nil
0
cl-user(4): 

The gid, initgroups-user, and uid arguments are independent and are processed in the following order, using the indicated system calls (on most UNIX platforms):

gid setgid(), setegid()
initgroups-user initgroups()
uid setuid(), seteuid()

The group is always set first, since after changing users that user may not have permission to change groups.

16. A couple of Unix examples

;; The current values of the environment variables on your system may,
;; of course, be different from what appears in this example.
user(2): (sys:getenv "SHELL")
"/bin/csh"
user(3): (setf (sys:getenv "SHELL") "/bin/sh")
"/bin/sh"
user(4): (sys:getenv "SHELL")
"/bin/sh"

Here are a couple of examples of run-shell-command. In the first, we simply have run-shell-command execute a simple command (who).

USER(1): (run-shell-command "who")
rlogin ttyb Aug 19 08:26
sdj ttyp0 Aug 18 16:08 (rubix)
adam ttyp2 Aug 18 16:17 (rubix)
dm ttyp4 Aug 19 10:24 (rubix)
0
USER(2):

The second example is more complicated. We cause run-shell-command to spawn a shell and then we communicate with the shell via the stream set up by run-shell-command.

;; First we define a function to read the output from the shell. This
;; function is pretty simple -- it reads characters and prints them
;; out but it does show how a more useful function could be implemented.
USER(24): (defun get-from-shell (stream)
             (do ((ch (read-char-no-hang stream)
                      (read-char-no-hang stream)))
                 ((null ch))
               (write-char ch)))
GET-FROM-SHELL
;; Now we initiate the shell:
USER(25): (setq shell-stream 
                (excl:run-shell-command "csh"
                                        :input :stream
                                        :output :stream
                                        :wait nil))
#<EXCL::BIDIRECTIONAL-TERMINAL-STREAM @ #x10a4aa6>
USER(26): (format shell-stream "who~%")
NIL
USER(27): (force-output shell-stream)
NIL
USER(28): (get-from-shell shell-stream)
rlogin ttya Aug 19 07:06
rlogin ttyb Aug 19 08:26
sdj ttyp0 Aug 18 16:08 (rubix)
cheetham ttyp1 Aug 18 17:17 (frozen)
adam ttyp2 Aug 18 16:17 (rubix)
NIL
;; We exit the shell:
USER(29): (format shell-stream "exit~%")
NIL
;; and close the stream.
USER(30): (close shell-stream)
T
;; We call sys:reap-os-subprocess because we called 
;; run-shell-command with :wait nil:
USER(31): (sys:reap-os-subprocess)
0
3995
nil
USER(32):

17A. excl:run-shell-command and SMP

A call to run-shell-command with :wait t (the default for the wait keyword argument is t) in an SMP Lisp blocks the Lisp process than calls run-shell-command but does not block other running processed. This is in contrast to the behavior of non-SMP multiprocessing, where a call to run-shell-command with :wait t blocks everything until the shell command completes, as we describe under the next heading.

17B. excl:run-shell-command and non-SMP multiprocessing

run-shell-command does not take multiprocessing into consideration. Therefore, if it is called with the :wait argument true (the default is t), all of Lisp waits for the call to complete, not just the process or thread that called run-shell-command. It is that behavior which is multiprocessing unfriendly. The following is a multiprocessing friendly call to run-shell-command. It does cause the calling process or thread to wait but does not cause the entire Lisp process to wait for the shell command to finish.

(multiple-value-bind (s errs my-pid)
    (run-shell-command "sleep 5; ls /usr/bin" :wait nil)
  (declare (ignore errs s))
  (let ((my-status nil))
    (mp::process-wait "for run-shell-command to finish"
                      #'(lambda ()
                         (setq my-status
                          (or my-status
                             (sys:reap-os-subprocess
                              :pid my-pid :wait nil)))))
    my-status)) 

Notes:

  1. Calling run-shell-command with :wait nil allows Lisp to continue in any case, and that might be what you want.
  2. To repeat what we said above, most things can be done better from Lisp without recourse to run-shell-command, which is inherently risky (to some extent, it puts the fate of the Lisp process in the hands of a non-Lisp program over which Lisp may not have control). Whenever you are tempted to use run-shell-command, consider performing the same action within Lisp (perhaps using foreign functions).

18. A note on the order of execution when reading from a program to be reaped in a non-multiprocessing environment

Code written similar to the following skeleton of code may hang:

(multiple-value-bind (shell-stream error-stream process)
    (excl:run-shell-command cmd
       :input :stream :output :stream :error-output :stream)

  (when process
    (loop (when (sys:reap-os-subprocess :pid process :wait nil)
	     (return))))

  ;; now read from shell-stream and then close the streams

  )

In the code sample, the process is reaped prior to reading the process output. While this often works, because many programs don't bother to wait for all of their writes to complete before exiting, it may cause hanging if the pipe to which the data is sent fills up and thus not all data can be written until some reading is done, or the child program waits until each input has been read before writing more data. Some operating systems will cause select() to not return ready status on the output descriptor if any data at all is in that pipe, regardless of whether a call to write() would have succeeded.

The correct outline for the code is:

(multiple-value-bind (shell-stream error-stream process)
    (excl:run-shell-command cmd
       :input :stream :output :stream :error-output :stream)

  ;; do all the reading from shell-stream

  (when process
    (loop (when (sys:reap-os-subprocess :pid process :wait nil)
	     (return))))

  ;;  close the streams  
)

When you are using multiprocessing, you can use multiprocessing tools such as process-wait to ensure that no hanging occurs, as is done in the example under the heading excl:run-shell-command and multiprocessing above.

19. Killing a process started with :wait nil

Such processes often exit on there own or are killed by the user of the application but you may wish to kill them yourself from within Lisp. To do this, you need the pid value (which is the third returned value from run-shell-command when :wait is nil). We assume in what follows that the pid value is pid. Then:

Killing a :wait nil process on UNIX

The function in the excl.osi package kill takes pid and a kill signal as arguments. For example,

(multiple-value-setq (v1 v2 xterm-pid)
   (run-shell-command "exec xterm" :wait nil))
nil
nil
3187
;; An xterm appears. The pid is 3187, which is the value of XTERM-PID

;; Later: (excl.osi:kill 3187 excl::*sigkill*) 
;; excl::*sigkill* is the constant associated with that signal 
;; The xterm is killed.  
;;
;; ALWAYS reap the killed process 
(sys:reap-os-subprocess :pid 3187)
0
3187
9

Note the call to sys:reap-os-subprocess. This cleans the process from the system. If you neglkect to do this and you start many subprocesses, the system will clog up.

This method does not work on Windows (kill signals an error if called in a Windows Lisp.

Killing a :wait nil process on Windows

This is a bit more complicated because:

In light of these differrences, be sure to test potential uses of killing spawned processes to ensure that what you want dome is actually done.

The following form kills the process with pid pid:

 (run-shell-command (format nil "taskkill /f /pid ~a" pid))

So, for example, we start notepad and then kill it:

(multiple-value-setq (v1 v2 pid)
  (run-shell-command "notepad" :wait nil))

;; Notepad runs

(run-shell-command (format nil "taskkill /f /pid ~a" pid))
;; Notepad is killed

(sys:reap-os-subprocess :pip pid)
;; we reap the process

See also shell. See os-interface.htm for general information on the interface between Allegro CL and the operating system and for information on shell commands.


Copyright (c) 1998-2019, Franz Inc. Oakland, CA., USA. All rights reserved.
This page was not revised from the 8.2 page.
Created 2012.5.30.

ToCDocOverviewCGDocRelNotesFAQIndexPermutedIndex
Allegro CL version 9.0
Moderate update since 9.0 release.
8.2 version