{"id":186,"date":"2020-08-18T19:23:47","date_gmt":"2020-08-18T20:23:47","guid":{"rendered":"http:\/\/www.linux-tutorial.info\/?page_id=77"},"modified":"2020-08-22T19:26:18","modified_gmt":"2020-08-22T20:26:18","slug":"this-is-the-page-title-toplevel-20","status":"publish","type":"page","link":"http:\/\/www.linux-tutorial.info\/?page_id=186","title":{"rendered":"Pipes and Redirection"},"content":{"rendered":"\nPipes and Redirection<\/title>\n<question id=\"43\" text=\"What symbol is used to 'pipe' two commands together?\" \/>\n<question id=\"44\" text=\"Pipes can be used to send the output of one command to another.\" \/>\n<question id=\"45\" text=\"The output of commands can be sent to files using what symbol?\" \/>\n<concept id=\"40\" description=\"The output of one command can be sent to another using the pipe symbol - |.\" \/>\n<concept id=\"43\" description=\"Multiple pipes can be used to combine multiple commands.\" \/>\n<p>\nPerhaps the most commonly used character is “|”, which is referred to as the\npipe symbol, or simply <glossary>pipe<\/glossary>.\n This enables you to pass the output of one command\nthrough the input of another. For example, say you would like to do a long\ndirectory listing of the <directory>\/bin<\/directory> directory. If you type\nls -l and then press Enter,\nthe names flash by much too fast for you to read. When the display finally\nstops, all you see is the last twenty entries or so.\n<\/p>\n<question id=\"\" type=\"mc\" text=\"What is the process of sending the output of one command to the input of another?\" \/>\n<p>\nIf instead we ran the command\n<commandexample command=\"more\">ls -l | more<\/commandexample>\nthe output of the <command>ls<\/command> command is said to be\n“piped through more”. The process is called “piping”.\nIn this way, we can scan through the list a screenful at a time.\n<\/p>\n<p>\nIn our discussion of <glossary>standard input<\/glossary>\nand <glossary>standard output<\/glossary> in the section on\n<site id=\"4\">basic operating system concepts<\/site>,\nwe talked about <glossary>standard input<\/glossary>\nas being just a file that <em>usually<\/em> points to your terminal.\nIn this case, standard output\nis also a file that usually points to your <glossary>terminal<\/glossary>.\nThe <glossary>standard output<\/glossary>\nof the ls command is changed to point to the\npipe, and the <glossary>standard input<\/glossary>\nof the more command is changed to point to the pipe as well.\n<\/p>\n<concept id=\"\" description=\"A pipe takes up physical space on the hard disk.\" \/>\n<question id=\"\" type=\"tf\" text=\"A pipe takes up physical space on the hard disk.\" \/>\n<p>\nThe way this works is that when the <glossary>shell<\/glossary>\nsees the <glossary>pipe<\/glossary> symbol, it creates a\ntemporary file on the hard disk. Although it does not have a name or directory\nentry, it takes up physical space on the hard disk. Because both the terminal\nand the <glossary>pipe<\/glossary>\nare seen as files from the perspective of the <glossary>operating system<\/glossary>,\nwe are saying that the respective commands should use different files instead of standard input and <glossary>standard output<\/glossary>.\n<\/p>\n<question id=\"47\" text=\"What three files are opened by default when a shell starts?\" \/>\n<concept id=\"54\" description=\"Three files are opened by the shell when your start it- standard input, standard output and standard error.\" \/>\n<p>\nWhen you log in and are working from the <glossary>command line<\/glossary>,\n<glossary>standard input<\/glossary>\nis taken from your <glossary>terminal<\/glossary>\nkeyboard and both standard output and <glossary>standard error<\/glossary>\nare sent to your <glossary>terminal<\/glossary> screen. In other words, the\nshell expects to be getting its input from the keyboard and showing the output\n(and any error messages) on the <glossary>terminal<\/glossary> screen. This could be a\nphysical terminal directly connected to the machine or a pseudo-terminal when you connect from a remote machine.\n<\/p>\n<concept id=\"\" description=\"Stdin, stdout, and stderr are references to files that the shell automatically opens when it starts.\" \/>\n<p>\nActually, the three (standard input, <glossary>standard output<\/glossary>,\n and standard error)\nare references to files that the <glossary>shell<\/glossary>\nautomatically opens. Remember that in\nUNIX, everything is treated as a file. When the <glossary>shell<\/glossary>\nstarts, the three files it\nopens are usually the ones pointing to your <glossary>terminal<\/glossary>.\n<\/p>\n<p>\nWhen we run a command like <command>cat<\/command>, it gets input from a\nfile that it displays\nto the screen. Although it may appear that the <glossary>standard input<\/glossary>\nis coming from that file, the <glossary>standard input<\/glossary>\n(referred to as stdin) is still the keyboard. This\nis why when the file is large enough and you are using something like\n<command>more<\/command> to display\nthe file one screen at a time and it stops after each page, you can\ncontinue by pressing either the Spacebar or Enter key. That’s because\n<i>standard <\/i>input is still the keyboard.\n<\/p>\n<p>\nAs it is running, more is displaying the contents of the file to the screen.\nThat is, it is going to <glossary>standard output<\/glossary>\n(stdout). If you try to do a more on a file that does not exist, the message\n<\/p>\n<p>\n<screenoutput>file_name: No such file or directory<\/screenoutput>\n<\/p>\n<p>\nshows up on your <glossary>terminal<\/glossary>\nscreen as well. However, although it appears to be\nin the same place, the <glossary>error message<\/glossary>\nwas written to <glossary>standard error<\/glossary>\n(stderr). (I’ll show how this differs shortly.)\n<\/p>\n<\/p>\nWhen you start a command, these three standard files are inherited by the new process. So, this command will take input from the keyboard and send output and error messages\nto the terminal screen. When you input a command line containing a <glossary>pipe<\/glossary>, the system know to change these for the respective commands.\n<p>\n<p>\nWriting to and reading from pipes is not the only way the system can change stdin, stdout and stderr. You can also use real files for redirection. This is done quite often using a pair of characters: <keyinput><<\/keyinput> and <keyinput>><\/keyinput>.\nThe more common of the two, “>,” redirects the\n<em>output<\/em> of a command into a file. That is, it changes\n<glossary>standard output<\/glossary>.\nAn example of this would be <keyinput>ls \/bin > myfile<keyinput>. If we were to run this command, we would\nhave a file (in my current directory) named myfile that contained the output of\nthe ls \/bin command. This is because <glossary>stdout<\/glossary>\nis the file myfile and not the\nterminal. Once the command completes, <glossary>stdout<\/glossary>\nreturns to being the <glossary>terminal<\/glossary>.\nWhat this looks like graphically, we see in the figure below.\n<\/p>\n<p>\n<div align=\"center\">\n<img decoding=\"async\" src=\"redirect.png\" width=445 HEIGHT=429 BORDER=0 usemap=\"#redirect_map\">\n<\/div>\n<map name=\"redirect_map\">\n<area shape=\"RECT\" coords=\"2,34,160,64\" href=\"popup#Standard Input#By default, standard input (stdin) comes from your keyboard.\">\n<area shape=\"RECT\" coords=\"169,2,298,60\" href=\"popup#Standard Input#By default, standard input (stdin) comes from your keyboard.\">\n<area shape=\"RECT\" coords=\"0,79,166,99\" href=\"popup#Standard Output#By default, standard output (stdout) is sent to your monitor screen.\">\n<area shape=\"RECT\" coords=\"0,119,167,142\" href=\"popup#Standard Error#By default, standard error (stderr) is sent to your monitor screen.\">\n<area shape=\"RECT\" coords=\"176,69,288,160\" href=\"popup#Standard Output & Error#By default, standard output (stdout) and standard error (stderr) are sent to your monitor screen.\">\n<area shape=\"RECT\" coords=\"3,251,301,311\" HREF=\"popup#Standard Input#After the re-direction, standard input <i>still<\/i> comes from the keyboard.”>\n<area shape=\"RECT\" coords=\"2,323,177,352\" HREF=\"popup#Standard Output#After the re-direction, standard output <i>still<\/i> goes to your monitor screen.”>\n<area shape=\"RECT\" coords=\"183,316,297,409\" HREF=\"popup#Standard Output & Error#After the re-direction <i>only<\/i> standard output appears on your monitor.”>\n<area shape=\"RECT\" coords=\"3,368,168,427\" href=\"popup#Standard Error#Standard error (stderr) has been redirected and no longer appears on you monitor screen.\">\n<area shape=\"RECT\" coords=\"327,369,444,426\" href=\"popup#Standard Error#Standard error (stderr) no longer appears on your monitor screen, but is sent instead to the file error.file.\">\n<\/map>\n<p>\nNow, we want to see the contents of the file. We could simply say\n<commandexample command=\"more\">more myfile<\/commandexample>, but that wouldn’t explain about <glossary>redirection<\/glossary>. Instead, we input\n<p>\n<commandexample command=\"more\">more <myfile<\/commandexample>\n<p>\nThis tells the more command to take its <glossary>standard input<\/glossary>\nfrom the file myfile instead of from the keyboard or some other file. (Remember, even when\nstdin is the keyboard, it is still seen as a file.)\n<\/p>\n<p>\nWhat about errors? As I mentioned, <glossary>stderr<\/glossary>\n<em>appears<\/em> to be going to\nthe same place as <glossary>stdout<\/glossary>.\n A quick way of showing that it doesn’t is by using\noutput <glossary>redirection<\/glossary>\nand forcing an error. If wanted to list two directories and\nhave the output go to a file, we run this command:\n<\/p>\n<p>\n<commandexample command=\"ls\">ls \/bin \/jimmo > \/tmp\/junk<\/commandexample>\n<\/p>\n<p>\nWe then get this message:<\/p>\n<p>\n<screenoutput>\/jimmo not found <\/screenoutput>\n<\/p>\n<question id=\"\" type=\"mc\" text=\"Assuming you are starting a command with a lot of output. All of the output should go into one file, all errors into another. What command would you issue?\" \/>\n<p>\nHowever, if we look in <directory>\/tmp<\/directory>, there is indeed a file called junk that\ncontains the output of the <command>ls<\/command> <directory>\/bin<\/directory> portion of the command. What happened here\nwas that we redirected <glossary>stdout<\/glossary>\ninto the file <file type=\"\">\/tmp\/junk<\/file>. It did this with the\nlisting of <directory>\/bin<\/directory>. However, because there was no directory <directory>\/jimmo<\/directory> (at least not on\nmy system), we got the error <directory>\/jimmo<\/directory> not found. In other words, <glossary>stdout<\/glossary>\nwent into the file, but <glossary>stderr<\/glossary>\nstill went to the screen.\n<\/p>\n<p>\nIf we want to get the output and any error messages to go to the same place,\nwe can do that. Using the same example with ls, the command would be:\n<\/p>\n<p>\n<commandexample command=\"ls\">ls \/bin \/jimmo > \/tmp\/junk 2>&1<\/commandexample>\n<\/p>\n<concept id=\"48\" description=\"Error messages can be sent to a file using the construct '2> file_name'” \/>\n<question id=\"\" type=\"mc\" text=\"What construct would send error message to the file \/tmp\/errors?\" \/>\n<question id=\"46\" text=\"2>&1 at the end of a command does what?” \/>\n<question id=\"\" type=\"mc\" text=\"What command would send the output of a command to 'nowhere' so that it does not appear on the screen? \" \/>\n<question id=\"\" type=\"mc\" text=\"What command would send the error output of a command to 'nowhere' so that it does not appear on the screen? \" \/>\n<concept id=\"\" description=\"You can send the output of a command to \/dev\/null so that it simply 'disappears'.\" \/>\n<question id=\"\" type=\"TF\" text=\"You can send the output of a command to \/dev\/null so that it simply 'disappears'.\" \/>\n<p>\nThe new part of the command is 2>&1, which says that file descriptor\n2 (stderr) should go to the same place as file <glossary>descriptor<\/glossary>\n1 (stdout). By changing the command slightly like this:\n<\/p>\n<p>\n<commandexample command=\"ls\">\nls \/bin \/jimmo > \/tmp\/junk 2>\/tmp\/errors<\/commandexample>\n<\/p>\n<p>\nwe can tell the <glossary>shell<\/glossary>\nto send any errors someplace else. You will find quite\noften in <glossary>shell<\/glossary>\nscripts throughout the system that the file that error messages\nare sent to is <device>\/dev\/null<\/device>. This has the effect of ignoring the messages\ncompletely. They are neither displayed on the screen nor sent to a file.\n<\/p>\n<p>\nNote that this command does not work as you would think:\n<\/p>\n<p>\n<commandexample command=\"ls\">ls \/bin \/jimmo 2>&1 > \/tmp\/junk<\/commandexample>\n<\/p>\n<concept id=\"41\" description=\"You can redirect error messages from commands to a file with 2>file_name.” \/>\n<p>\nThe reason is that we redirect <glossary>stderr<\/glossary>\nto the same place as <glossary>stdout<\/glossary>\n<em>before\n<\/em>we redirect <glossary>stdout<\/glossary>.\n So, <glossary>stderr<\/glossary>\ngoes to the screen, but stdout goes to the\nfile specified.\n<\/p>\n<concept id=\"42\" description=\"You can combine pipes and redirection in a single command.\" \/>\n<p>\nRedirection can also be combined with pipes like this:\n<\/p>\n<p>\n<commandexample command=\"sort\">sort < names | head <\/commandexample>\n<\/p>\n<p>\nor\n<\/p>\n<p>\n<commandexample command=\"ps\">ps | grep sh > ps.save<\/commandexample>\n<\/p>\n<p>\nIn the first example, the <glossary>standard input<\/glossary>\nof the <command>sort<\/command> command is redirected\nto point to the file names. Its output is then passed to the <glossary>pipe<\/glossary>.\n The standard\ninput of the head command (which takes the first ten lines) also comes from the\n<glossary>pipe<\/glossary>. This would be the same as the command\n<\/p>\n<p>\n<commandexample command=\"sort\">sort names | head<\/commandexample>\n<\/p>\n<p>\nwhich we see here:<\/p>\n<p>\n<p>\n<div align=\"center\">\n<img decoding=\"async\" src=\"pipe.png\" width=352 HEIGHT=44 BORDER=0 usemap=\"#pipe_map\">\n<\/div>\n<map name=\"pipe_map\">\n<area shape=\"RECT\" coords=\"0,0,145,43\" href=\"popup#Standard output of the command#In this example, the standard output of the sort command is sent into the pipe.\">\n<area shape=\"RECT\" coords=\"146,0,264,42\" href=\"popup#Pipe#Depending on your version of Linux and how it is configured, the pipe may actually be a file on the hard disk, but could also only be in memory.\">\n<area shape=\"RECT\" coords=\"248,2,349,41\" href=\"popup#Standard input of the next command#The standard input for the head command comes from the pipe in this example.\">\n<\/map>\n<p>\nIn the second example, the <command>ps<\/command> command (process status) is piped through <command>grep<\/command> and all of the output is redirected to the file <file type=\"\">ps.save<\/file>.\n<\/p>\n<p>\nIf we want to redirect <glossary>stderr<\/glossary>, we can. The syntax is similar:<\/p>\n<p>\n<commandexample>command 2> file<\/commandexample>\n<\/p>\n<question id=\"48\" text=\"What symbol separates commands on a single line?\" \/>\n<p>\nIt’s possible to input multiple commands on the same <glossary>command line<\/glossary>.\nThis can be accomplished by using a semi-colon (;) between commands. I have used this on\noccasion to create command lines like this:\n<\/p>\n<p>\n<commandexample command=\"man\">man bash | col -b > man.tmp; vi man.tmp; rm man.tmp<\/commandexample>\n<\/p>\n<question id=\"\" type=\"mc\" text=\"What command could be used to send output to a file as well as to the screen?\" \/>\n<p>\nThis command redirects the output of the <glossary>man-page<\/glossary>\nfor bash into the file <file type=\"\">man.tmp<\/file>. (The <glossary>pipe<\/glossary>\nthrough <keyinput>col -b<keyinput> is necessary because of the way the man-pages\nare formatted.) Next, we are brought into the vi editor with the file <file type=\"\">man.tmp<\/file>.\nAfter I exit <command>vi<\/command>, the command continues and removes my temporary file <file type=\"\">man.tmp<\/file>.\n(After about the third time of doing this, it got pretty monotonous, so I\ncreated a shell script to do this for me. I’ll talk more about shell scripts\nlater.)\n<\/p>\n<p>\nSometimes it is useful to not only redirect the output of a command to a file, but to monitor the output at the same time. One way would be to use to screens. The first re-directs the output of the command to a file. On the second screen you could read the output file as it is being written by using <command name=\"tail\">tail -f<\/command>.\n<\/p>\n<question id=\"\" type=\"\" text=\"What command could be used to send output to both a file and standard out?\" \/>\n<concept id=\"\" description=\"The 'tee' command can be used to send output to both a file and standard out.\" \/>\n<p>\nAlternative you can use the the <command>tee<\/command> command. As its name implies, the <command>tee<\/command> command creates a “T” whereby the output is sent to a the file specified as well as to standout. For example:\n<p>\n<p>\n<commandexample command=\"tee\">sort input_file | tee output_file <\/commandexample>\n<\/p>\n<p>\nThis command would send the output of the <command>sort<\/command> to the file <keyinput>output_file<\/keyinput> and to the screen (i.e. standard out).\n<\/p>\n<p>\nNote that the output of <command>tee<\/command> is sent to standardout, it would therefore be possible to send the output of <command>tee<\/command> to a second pipe. If fact, you use <command>tee<\/command> multiple times with multiple files:\n<\/p>\n<commandexample command=\"tee\">sort input_file | tee \/tmp\/output_file1 | tee \/tmp\/output_file2 | tee \/tmp\/output_file3<\/commandexample>\n","protected":false},"excerpt":{"rendered":"<p>Pipes and Redirection Perhaps the most commonly used character is “|”, which is referred to as the pipe symbol, or simply pipe. This enables you to pass the output of one command through the input of another. For example, say … <a href=\"http:\/\/www.linux-tutorial.info\/?page_id=186\">Continue reading <span class=\"meta-nav\">→<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-186","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"http:\/\/www.linux-tutorial.info\/index.php?rest_route=\/wp\/v2\/pages\/186","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.linux-tutorial.info\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"http:\/\/www.linux-tutorial.info\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"http:\/\/www.linux-tutorial.info\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.linux-tutorial.info\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=186"}],"version-history":[{"count":1,"href":"http:\/\/www.linux-tutorial.info\/index.php?rest_route=\/wp\/v2\/pages\/186\/revisions"}],"predecessor-version":[{"id":658,"href":"http:\/\/www.linux-tutorial.info\/index.php?rest_route=\/wp\/v2\/pages\/186\/revisions\/658"}],"wp:attachment":[{"href":"http:\/\/www.linux-tutorial.info\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=186"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}