CGI, Perl, SSI, etc.

From The Common Gateway Interface:

The Common Gateway Interface (CGI) is a standard for interfacing external applications with information servers, such as HTTP or Web servers. A plain HTML document that the Web [server] retrieves is static, which means it exists in a constant state; a text file that doesn't change. A CGI program, on the other hand, is executed in real-time, so that it can output dynamic information.

Most CGIs you find on the web are probably written in Perl. For more information on Perl, see "What is Perl?".

For more specific information about CGI, and for CGI programs and scripts, go to "The CGI Resource Index" or "The Common Gateway Interface" or the "KnitWit explains CGI Scripts for free.prohosting.com" page.

A lot. You can create CGIs to do any number of things, including, but not limited to: guestbooks, threaded message boards, banner ad placement, random links and images in web pages, file uploading, form processing and much more. You just need to know how to write them, or get them from somewhere.

I'm not going to even try to get into how to do any of this here. Just keep reading...

If you need help writing CGI or Perl script code is correct, DO NOT send questions to me about it! I don't know enough about CGI/Perl programming or other programming languages to help you. I don't know anything about Perl Modules or MLDBM or TCL or any other CGI stuff that you might want to know about. If you want to try different things on the server to see if they will work, and you think that others might be interested in them, feel free to let me know about them and I'll add them to the FAQ. But beyond what's written in this FAQ, questions about this stuff sent to me will be deleted without a reply. You should try posting the question to a CGI or Perl newsgroup such as "comp.infosystems.www.authoring.cgi", or some other newsgroup or mailing list related to your question(s).

The "cgi-bin" is the directory that you put all your CGI programs and scripts. The directory permissions need to be "chmod 755" (user = read/write/execute, group/other = read/execute).

The "cgi-bin" is the sandbox in which CGIs play. If CGIs are placed outside of this directory, then calls to them will result in errors because the server will not let them run.

If you do not have a "cgi-bin" directory in your "html" directory, then you will have to create one and set the permissions accordingly. See the Permissions Guide for more information.

No. That's how ProHosting set things up.

Yes. That's how ProHosting set things up.

You need to know your home directory is on the server to be able to know where to point to files within your CGI scripts. To find this out, you should go to the Account Administration page and log in, and it will tell you.

You can also try this script. Save it with a ".cgi" extension, upload it to your cgi-bin directory and it'll print out the path to your cgi-bin directory.

There are 2 types of paths: URL paths and system paths.

URL paths are what you use in your web pages to refer to images and other pages. The paths to the files are relevent only to the web server.

System paths are paths that refer to files on the web server's file system. These mean nothing to the web server, but they are needed when referring to files in a CGI.

Some examples:

• URL Paths:

  To point to a script in the "ACTION=" attribute in a FORM tags and SRC attributes in IMG tags, use a URL path:

  /~username/cgi-bin/script.pl
       or
  /~username/images/pic2.gif
  
  
• System Paths:

  To reference files from inside the CGI (for example, files to read from and write to), use:

  /usr/home#/username/html/location/filename.ext *
       or
  /usr/home/web/u/username/location/filename.ext *

* Using system paths, you cannot access files on another server. For info on "home" directories, see "Why are their different "home" directories?"

Perl is a script-based programming language or scripting language. You can use Perl to write CGIs. Perl scripts are simply text files (upload it using ASCII mode, no compiling needed) and the extension for the file can be either ".pl" or ".cgi". Perl scripts must be put in the cgi-bin directory in your user directory, and the permissions set to "chmod 755" (user = read/write/execute, group/other = read/execute). See the Permissions Guide for more information.

I will not go into the art of Perl programming here. There are lots of pre-configured Perl scripts that perform a variety of functions, from hit counters to guestbooks to rotating banner insertions, which can be found all over the Internet. If you want to learn to program in Perl, buy a book (O'Reilly puts out some good ones), or visit Perl Mongers (formerly The Perl Institute) or www.perl.com

It means that it is not compiled ahead of time, but is compiled at run-time (or run through in an interpreter). In the case of Perl, the script is run by the Perl interpreter on the server when the script is called. For the most part, Perl scripts are the easiest ones you can use, and are probably the most common ones you'll find.

Okay. I decided to put together a small sample script, written in Perl, which performs a few common CGI tasks. The items that are demonstrated include:

• Using variables
• Using environment variables
• Retrieving query string data
• Reading text from a file
• Processing data from HTTP forms
• Writing data to a file

Click on the link below to display the script in a new window. What you see is the results of running the script, which shows you some things you can do. At the top of the page is a link to a plain text file that contains the source of the script, which you can download and use at your leisure. Reading through the script source will tell you a lot, because the whole script is filled with comments describing nearly every line of code.

Yes, so long as you aren't doing anything in the script that is platform specific (meaning things that one operating system can do that others can't). You will want to make sure that you have directories set up the same way as on the server or you'll have to change any directory paths in the script to reflect the directory structure on the server before uploading.

If you are using Windows 95/98/NT/2000, you can find "Perl for Win32" at ActiveState. For Mac users, get "MacPerl" or "MPW tool" at The MacPerl Pages. Most Unix systems come with Perl built in (an most Unix users already know about Perl anyway). If you need Perl for Unix, go to Perl Mongers and look there.

You can use Perl with most web servers, including Personal Web Server for Windows 95/98/NT/2000 or Apache for Unix or Windows. It's a good idea to set up a small server on your PC to test with before uploading to ProHosting.

No. Not as far as ProHosting is concerned, anyway.

You can't. At least not on ProHosting's servers.

Currently, I believe they are using Perl version 5.0. This is important to know because Perl 5.0 has more, and in some cases, different features then other versions of Perl (although it is backwards compatible with earlier versions).

The Perl interpreter is located in at:

To reference Perl in your scripts, at the top of your perl scripts should be the line one of those lines (either one should work). This may be different on other machines, so if you download scripts from some other location, make sure the first line in the script (you can edit them in any text editor) is as above. What this does is tell the web server where to find the Perl interpreter that takes the script, reads the code, processes it and spits back whatever the results are.

CGIs are available from all over the web. Some are free, some are shareware (or postcardware or e-mailware or some such thing). Perl scripts are often found for free. You can visit "Perl Mongers" for more information and the history of Perl.

There are lots of CGIs and Perl scripts out there. You can get some from "The CGI Resource Index" and "KnitWit explains CGI Scripts for free.prohosting.com". There are also lots to be found at "Perl Mongers" or "www.perl.com". Or search for "CGI" or "Perl" at your favorite search engine.

StopBy has posted a number of scripts modified to work for ProHosting, including scripts for file management and editing.

NOTE: When you download a script from someone's web site, it often comes with instructions on how to configure the script to work. Often this involves changing the paths and a few other variables near the top of the script file. Be aware that some of the instructions for the script, in particular the ones relating to permissions, are not likely to be correct for use at ProHosting. Most often, the instructions tell you the permissions must be 777. See the Permissions Guide for more information.

Go to David Efflandt's page and download and install the "findpmod.cgi" script. It is a Perl Module Search Script. Upload the script to your "cgi-bin" directory and use your browser to view it. The script will let you search for a Perl module, and if it exists, it will give you the path to the module and a link to the manpage (help file) for it.

To use this, you will need to know about Perl modules and how to use them. If anyone has a good, concise explanation of them, please let me know.

You won't work with CGI for too long before you run into words like GET, POST and HEAD. These are HyperText Transfer Protocol methods (commands). HTTP is the protocol that is used to transfer files from a web server to a web browser via the Internet. GET and POST are the methods that are most used. These commands are case-sensitive, so GET is not the same as get or Get.

HTTP requests and the responses from the server usually consist of two parts: the header and the body. Request headers can contain the method (GET, POST, etc.), file requested, browser type, connection type, and what type of content that will be accepted in the reply; the request body can contain URL-encoded data (see next question). Reply headers can contain status information, time stamps, server info, the type and length of the body to follow, with the reply body containing the contents of the file or the output of a CGI.

Forms in your HTML pages can submit form data to a CGI with either GET or POST. That's what the "METHOD=" attribute of the FORM tag is for. URLs in your web browser's location bar and HREF links in HTML pages use the GET method.

• GET:

  The GET method is used to request information from the server. GET requests get a result from the web server that come from a file, the result of a CGI output or some other source. There is no body part to a GET request. The request can contain data in the "QUERY_STRING", which is made up of key/value pairs that appear in the URL like this: /path/script.cgi?key1=value1&key2=value2. Anything after the "?" is part of the "QUERY_STRING", which can be read by a CGI.

  
  
• POST:

  The POST method is used to give larger amounts of data to a CGI on the server. In a POST request, form data is formatted the same as in the "QUERY_STRING", but it comes in the body of the request instead of tacked on to the URL. The header of a POST request will contain the length (in bytes) of the body content to follow, e.g. Content-length: 54, which you can use to determine of any data was sent.

  
  
• HEAD:

  The HEAD method is pretty much the same as GET. However, it does not return the body of the result, only the header information. This is useful for seeing if files are present and how big they are, without haven't to wait for the whole file to be sent. You wouldn't use this for HTML forms, but you might use it in a link-checking CGI.

99.9% of the time, GET and POST are the only ones you'll use. GET, POST and HEAD are all part of the HTTP 1.0 standard. There are others that are defined in the HTTP 1.1 standard, such as PUT and DELETE. I'm not going to explain them here because you will most likely never use them.

As far as I know, PUT and DELETE are not supported at ProHosting. In fact, you'd be hard pressed to find a server on the Internet that both supports these other methods and has them enabled (at least for the general public). A lot of web servers don't actually support all the methods defined in the standard for one reason or another, and if they do support them, they are disabled for security reasons.

URL encoding is a way to encode certain characters that a web server or web browser normally can't deal with. In the data that is sent to a web server, and on to the script as well, as part of the GET or POST methods, the data cannot contain certain characters, such as spaces ( ), ampersands (&), percent signs (%), and equals signs (=), amongst others. These types of characters are automatically encoded by the web browser into a format that has a % sign followed by an ASCII character code in hexadecimal notation, e.g. %3D is an =, %25 is an %, %20 is a space.

When you write a CGI script, some languages have built-in methods for URL encoding and decoding. Some do not. Perl does not, however there is a Perl module called CGI.pm that contains subroutines to do this decoding for you. If you just want to include that code in your script, you can use this:

This code is essentially the same as that in CGI.pm. It can handle both GET and POST methods. It creates an associative array called "formdata" which contains all the key/value pairs from your form data. Just paste this code in your script and when you want to parse the data, just use "&Parse_Form" to call the subroutine.

This is probably a problem with the permission settings on the CGI or the "cgi-bin" directory. You'll need to change the permissions to "755". See the Permissions Guide for more information.

This could be caused by any number of problems. Here's a list of the 6 most common problems people have, in no particular order.

The "shabang" is the first line in the script that starts with #!. # = sharp, ! = bang... programming lingo, don't worry about it. This tells the server where the Perl interpreter is located. For ProHosting, the shabang line needs to be:

This may be different for other systems, so you need to check to make sure it's correct if you copied the script from someone else.

Presumably, your script will return something to be displayed in the browser. If so, before any other information is returned, there needs to be this line:

This line has two (2) \n characters. The \n means "new line", and is technically only one character. It's one of a number of escape characters. The first new line ends the line and the second is a blank line, which tells the browser that header information is complete and the rest of the content is the body, which is what will be displayed in the browser. This is absolutely necessary. It does not need to come first after the shabang line, but it does need to come before any other print statements.

Also, if you're sending another type of data, you need to return the correct content type string. For example, a script returning JPEG image data, would use the content type "image/jpeg" instead of "text/html".

There are two types of paths to files on the server. There are URL paths, which are the paths that are used in the HTML to point to files or in the browser to point to a page. And there are system paths, which are used to point to files relative to the server. It is important to know the difference. See "How do I point to a CGI? How do I point to files from the CGI?" for the proper types of paths to use.

Remember: Scripts and directories should be 755, other related files should be 644. See the Permissions Guide for more information.

Remember to upload your scripts in ASCII mode, not binary. This will definitely mess your files up. See "What's the difference between ASCII and Binary transfer modes?".

All CGI/Perl scripts must be in a directory called "cgi-bin". Only the scripts need to be there, other files can be anywhere else. See "What is the "cgi-bin"?".

If the CGI is written in C, C++ or some other compiled language, and you downloaded it from somewhere, you'll need the original source code for it to edit it. You can't edit programs after they've been compiled, and unless the program was compiled for ProHosting's servers, it won't likely run as-is. See "Can I create CGIs with something other then Perl?" and "How can I make executable files run?"

There is a 20 second time limit on scripts. This is because scripts that take a long time run are often doing things that consume a lot of the system's resources. Nearly any task that you would want to do with a CGI should not take more then 20 seconds. That's a long time for a computer to be working on one thing. Just keep this in mind when you're writing your CGIs.

If your script has an error, most sources will tell you that you should check the server log for error messages and other information. Unfortunately, we don't have access to the server logs. This is for a number of reasons. Security: the server logs contain lots of information about everyone's accounts. Server load: the server is hosts thousands of accounts and the logs are kept for the entire server. To download the server log files would take a long time since it's a big file, and to pull out the information on the server for a particular user via a CGI script would take a long time and lots of processing power as well. Creating individual logs for users is probably too time consuming.

So how do you live without the server log? Well, the first step to fixing errors is to read the previous question. It describes the six most common causes of CGI errors. If that doesn't work, you could find a program that will check your CGI code for syntax errors. Another thing to do is to start simple and print out messages that tell what the CGI is doing. Comment out blocks of code to make the script as small as possible, work out the bugs in what you have, and slowly add more sections of code back into the program. This way you only have to deal with small steps, making it easier to figure out what's wrong. When you're done, just take remove or comment out any debug message print statements.

SSI stands for "Server Side Includes". This means that the server includes stuff in an HTML page, whether it be another file's contents or the output of a CGI program or script, as instructed by the SSI directives. The server, before sending the page to the browser, scans the file for SSI directives and inserts the requested stuff in it's place in the file.

SSI will only work in files with the extensions ".shtml". The reason is that most people have ".html" or ".htm" files, but not all of them have SSI directives. However, the server would still have to search through all of them to find SSI directives because it can't know ahead of time if there are none since SSI directives are only processed when the server serves the page. So to make things faster this, if you want to use SSI directives, put use the ".shtml" extension. If you are not going to use SSI in pages, just use ".html" or ".htm", which saves the server processing time.

An example of including in HTML files would be if you had 10 pages that you wanted to have the same footer information, such as copyright and contact information, on all the pages. Without SSI, you would have to make all your HTML pages have the same stuff at the bottom and if you wanted to change this information, then you would have to change all 10 files. With SSI, you can include the same footer information (stored in a file) at the end of all the documents. Then when changes are needed, you only have change the one footer file.

You can also call to CGIs with SSI and have the results print out into the HTML page when it's loaded. This is a good way to include a counter or random ads or links in your page.

To use SSI, you need to use special SSI directive code in your HTML files to tell the server what to include (a file or CGI output). The SSI directive code is placed in the HTML where you want the stuff to show.

To include files, use the following include directives in your HTML pages:

To include CGI output, use:

* SSI will NOT work if the "#" character has a space before it.

For more detailed info on server-side includes, see the SSI Test page.

Apparently, sometimes the server is sending ".shtml" files as plain text, so all you see in the browser is the page's source. At least, you see the source in Netscape Navigator, but Internet Explorer shows the page properly, with the exception that any SSI directives don't work.

This is a server configuration problem, and it shouldn't be a problem with only your site, but for any account on the same server. You can fix this yourself by creating an .htaccess file in your "html" directory with the following lines:

The first line tells the server to associate ".shtml" files with the "text/html" mime-type. The second line tells the server to parse ".shtml" files for SSI directives.

First, it's not installed at ProHosting.

Fly is a program that manipulates images. It's most common use is probably for hit counters. It will take the number images that make up the number of hits and put them into one image file (and can do a lot of other things). This is often used on servers that don't allow SSI to display a hit counter.

You might be able to run it in your "cgi-bin" directory. It's a C program and is probably available pre-compiled for BSD OS 4.01. For more information on Fly, go to the University of Melbourne Fly site. I don't know how to use it, either to compile it or to call to it in a CGI, so DO NOT ask me.

sendmail is what it sounds: a program that sends e-mail. Unfortunately, sendmail is NOT AVAILABLE from the free sites because of past problems with spamming (more). Hopefully, sendmail can return soon, perhaps dealing with abusers on a case-by-case basis.

SMTP (Simply Mail Transport Protocol) is a server for sending outgoing e-mail. I'm sure ProHosting has one, but it is NOT accessible from the free servers.

Well, since you can't use sendmail, there are two ways that I know of to send e-mail via CGIs.

One is to go to Bravenet. They offer all sorts of free on-line services, including free form processing. You'll have to visit the site to find out how to set it up. There may be other sites that offer similar services, and I don't know how reliable the services are.

The second is to connect to an outside SMTP (Simply Mail Transport Protocol) server. Go to this page at TULARC and you'll find Perl script code that you can use to connect to a SMTP server to send mail. The trouble will be finding a SMTP server that can be accessed from the "free.prohosting.com domain". Most restrict access to use within the local domain to keep out spammers.

The date command is used to get the date and time on the server in a CGI script for things like guestbook entries and the like. The date command is located at:

To use "date", you need to execute the command and capture the output to a variable, like so:

NOTE: It's a `, not a ´ (a backtick, not single quote). It's on the key with the ~ (tilde). Also, there will be a "\n" (newline character) on the end of the date. You can remove this with the chomp() function.

So, now you have a string in "$Date" with the date in it! And you can use this like:

Which would print out this:

Sample script:

"whois" can be used for looking up domain names and other info from the Network Solutions WHOIS database. whois is located at:

Script example:

Where "$who" is a something to be searched for.

No. Server-side Java stuff is not supported at ProHosting.

For those that want to know what these things are... Java Servlets are like CGIs written in Java. Java Server Pages is like ASP or PHP (If you know what that is) using Java code. And Enterprise JavaBeans... well, you can look that up at the Java web site, because I don't know what it is, off the top of my head, and I don't want to research it.

Yes. Applets don't run on the server, they run in the browser of the person viewing the applet. Just make sure to FTP (upload) your applet class files in binary mode and set up your HTML correctly.

Officially, I'm told that it is supported on the free servers. But in reality, it appears this isn't true. It may work on some machines, but maybe not others.

If it's supported on the machine your account is on, it's PHP version 4.0 RC1, last I checked.

For those that don't know, PHP is for server-side scripting, similar in function to ASP, but the code language looks more like Perl. PHP code is placed in with the HTML in a web page, and the server parse the page and does what the code says when the page is loaded, similar to how SSI works.

When using PHP, you don't use an .html or .shtml extension, you use a .php extension. Also, PHP pages only need to have 644 permissions, not 755. You can find out all about PHP at the PHP web site.

To give you a brief introduction, here's some sample code from the PHP web site documentation, which you can use to test if your account can use it:

The results in a web page would simply be:

Of course, there's a lot more that you can do then that, like access databases, process forms and assorted other things. Try mine.

No. ProHosting's web server does not support ASP. ASP stands for Active Server Pages, and was developed by Microsoft to do server-side scripting by including script code in an HTML page. It's similar in function to PHP... but that's all I'm gonna say about it here, since it's not supported anyway.

SQL (Structured Query Language), which is a database protocol, and mSQL & MySQL which are implementations of SQL, are supported through the paid services of ProHosting, but not the free services. Database services are generally not cheap in any sense, either price-wise or processor-wise. If you need this type of service, then most likely, you won't find many web services offering them for free.

There are ways to access database files with CGIs, but I don't have any details on that.

Microsoft FrontPage Extensions are not supported on the free sites, but it are through ProHosting's paid service.